Skip to content

Conversation

@filbeofITK
Copy link
Contributor

Odb 2.5 is in beta currently but, it is going live soon and it is said to be as stable as 2.4 was, before the gcc upgrades broke it.
I included some needed headers to the plugins/cpp/model/include/model/cppentity.h and the model/include/model/buildaction.h which were missing, but previously the odb headers included them, so this error stayed silent.
I wrote an install script too, and made the docker image to use it.

@mcserep
Copy link
Collaborator

mcserep commented Apr 19, 2020

Please amend the CI script for Bionic accordingly which would verify that with these modifications CodeCompass can be built.

@mcserep mcserep added this to the Release Flash milestone Apr 19, 2020
@mcserep mcserep linked an issue Apr 19, 2020 that may be closed by this pull request
@whisperity
Copy link
Contributor

Also, we need to consider people who do not with to install anything non-package-manager driven to system-wide paths... the current script uses /usr/local which I'd rather not pollute with development stuff (it already contains plenty of vendor-specific stuff) that is not easily possible to uninstall.

@whisperity
Copy link
Contributor

Also, we need to consider people who do not with to install anything non-package-manager driven to system-wide paths... the current script uses /usr/local which I'd rather not pollute with development stuff (it already contains plenty of vendor-specific stuff) that is not easily possible to uninstall.

For this to fix, I needed to add ${ODB_INCLUDE_DIRS} to the target_include_directories here:

function(add_odb_library _name)
add_library(${_name} STATIC ${ARGN})
target_compile_options(${_name} PUBLIC -Wno-unknown-pragmas -fPIC)
target_link_libraries(${_name} ${ODB_LIBRARIES})
target_include_directories(${_name} PUBLIC
${CMAKE_SOURCE_DIR}/util/include
${CMAKE_SOURCE_DIR}/model/include
${CMAKE_CURRENT_SOURCE_DIR}/include
${CMAKE_CURRENT_BINARY_DIR}/include
${CMAKE_BINARY_DIR}/model/include)
endfunction(add_odb_library)

@whisperity whisperity added Kind: Critical 💥 Target: Developer environment Developer environment issues consist of CodeCompass or 3rd-party build tooling, configuration or CI. Target: Documentation and removed Kind: Bug ⚠️ labels Apr 24, 2020
@filbeofITK
Copy link
Contributor Author

I agree that the /usr/local isn't the best permanent solution, I'm going to add the destination as a mandatory option to the script.

.travis.yml Outdated
- odb
- libodb-dev
- gcc-7
- gcc-7-plugin-dev
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A useful note for Travis @filbeofITK , that the CI starts with a pre-built image, a similar concept like Docker images.
The selected image is defined by the os (in our case linux), the distro (in our case xenial or bionic) and the language configuration parameter (in our case cpp for the CI).

These images have a couple of software preinstalled, as described here:
https://docs.travis-ci.com/user/languages/cpp/

It says that Xenial ships with GCC 5.4.0 and Bionic ships with GCC 7.4.0. These are typically installed in /usr/local/, but of course you can still install any APT package (to usr/), if necessary.

The minimal image is also useful to check, because the cpp image is a superset of that, so e.g. curl is part of all images, there is no need to install it explicitly:
https://docs.travis-ci.com/user/languages/minimal-and-generic/

Minimizing additional dependency installation can boost up the CI job runtime.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks :)

Copy link
Contributor

@csordasmarton csordasmarton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have some small comments and please don't forget to squash your commits before this pull request getting merged.

filbeofITK added 2 commits May 1, 2020 10:33
odb has to be build before thrift.
Added the sudo configuration to the build2 toolchain.

Remove the the data removal parts from the Dockerfile.
@filbeofITK
Copy link
Contributor Author

Also, we need to consider people who do not with to install anything non-package-manager driven to system-wide paths... the current script uses /usr/local which I'd rather not pollute with development stuff (it already contains plenty of vendor-specific stuff) that is not easily possible to uninstall.

For this to fix, I needed to add ${ODB_INCLUDE_DIRS} to the target_include_directories here:

function(add_odb_library _name)
add_library(${_name} STATIC ${ARGN})
target_compile_options(${_name} PUBLIC -Wno-unknown-pragmas -fPIC)
target_link_libraries(${_name} ${ODB_LIBRARIES})
target_include_directories(${_name} PUBLIC
${CMAKE_SOURCE_DIR}/util/include
${CMAKE_SOURCE_DIR}/model/include
${CMAKE_CURRENT_SOURCE_DIR}/include
${CMAKE_CURRENT_BINARY_DIR}/include
${CMAKE_BINARY_DIR}/model/include)
endfunction(add_odb_library)

Could you help me out a bit please?
What else did you do, in order to help cmake find odb? I'm trying to export the path to the odb's .pc files, so when cmake does pkg_check_modules(PC_LIBODB QUIET "libodb") it works but it doesn't.

@whisperity
Copy link
Contributor

@filbeofITK GitHub's reply feature is stupid, it doesn't tag the person, please tag me to make sure I get the message!

I didn't do anything else. I have build2 downloaded and checked out at ~/Work/CodeCompass/Dependencies-Source. Same with Thrift and gtest.
The libraries on my local machine get installed to ~/Work/CodeCompass/Dependencies-Install. Everything goes to that directory. It look like this:

╰─ tree -L 2 $(pwd)
~/Work/CodeCompass/Dependencies-Install
├── bin
│  ├── odb
│  ├── odb.so
│  └── thrift
├── include
│  ├── cutl
│  ├── gmock
│  ├── gtest
│  ├── libstudxml
│  ├── odb
│  ├── sqlite3.h
│  ├── sqlite3ext.h
│  └── thrift
├── lib
│  ├── libcutl-1.11.0-b.7.so
│  ├── libcutl.so -> libcutl-1.11.0-b.7.so
│  ├── libgmock.a
│  ├── libgmock_main.a
│  ├── libgtest.a
│  ├── libgtest_main.a
│  ├── libodb-2.5.0-b.17.so
│  ├── libodb-sqlite-2.5.0-b.17.so
│  ├── libodb-sqlite.a
│  ├── libodb-sqlite.so -> libodb-sqlite-2.5.0-b.17.so
│  ├── libodb.a
│  ├── libodb.so -> libodb-2.5.0-b.17.so
│  ├── libsqlite3-1.so
│  ├── libsqlite3.a
│  ├── libsqlite3.so -> libsqlite3-1.so
│  ├── libstudxml-1.1.0-b.8.so
│  ├── libstudxml.so -> libstudxml-1.1.0-b.8.so
│  ├── libthrift-0.13.0.so
│  ├── libthrift.a
│  ├── libthrift.la
│  ├── libthrift.so -> libthrift-0.13.0.so
│  ├── libthrift_c_glib.a
│  ├── libthrift_c_glib.la
│  ├── libthrift_c_glib.so -> libthrift_c_glib.so.0.0.0
│  ├── libthrift_c_glib.so.0 -> libthrift_c_glib.so.0.0.0
│  ├── libthrift_c_glib.so.0.0.0
│  ├── libthriftz-0.13.0.so
│  ├── libthriftz.a
│  ├── libthriftz.la
│  ├── libthriftz.so -> libthriftz-0.13.0.so
│  └── pkgconfig
└── share
   └── doc

I built libodb-sqlite by explicitly specifying the import path for libodb:

(executed from ~/Work/CodeCompass/Dependencies-Source/build2, where I downloaded to and executed the build2-install-0.12.0.sh script)

bpkg create --quiet --jobs $(nproc) --wipe -d . cc \
  config.cxx=/usr/bin/g++                       \
  config.cc.coptions=-O3               \
  config.bin.rpath=$(readlink -f ../../Dependencies-Install/lib)      \
  config.install.root=$(readlink -f ../../Dependencies-Install) \
  config.import.libodb=libodb-2.5.0-b.17

Before creating the build directory for CodeCompass, I source in a script that sets the paths to the proper versions, as described in the documentation:

#!/bin/sh
CODECOMPASSS_DEPROOT="$(readlink -f ~/Work/CodeCompass/Dependencies-Install)"
LLVM_ROOT="$(readlink -f ~/Work/LLVM/Build)"

if [ -z "$(command -v ep)" ]
then
  echo "Fuck: No envprobe!" >&2
else
  ep ^CMAKE_PREFIX_PATH
  ep +CMAKE_PREFIX_PATH ${CODECOMPASS_DEPROOT} ${LLVM_ROOT}

  ep +PATH ${CODECOMPASS_DEPROOT}/bin ${LLVM_ROOT}/bin

  ep +LD_LIBRARY_PATH ${CODECOMPASS_DEPROOT}/lib ${LLVM_ROOT}/lib
fi

ep +PATH asdasd is a shorthand for export PATH="asdasd:${PATH}" basically.
So I add the "Dependencies-Install" tree (with bin and lib where appropriate) and the LLVM tree to the necessary paths. To be fair, LLVM is redundant because the build uses stock G++ 7.5.0 for current 18.04, and the LLVM 7.0 from the package manager as the library.

Created the build as follows:

╰─ cmake -DCMAKE_INSTALL_PREFIX="$(readlink -f ../Install)" -DDATABASE=sqlite -DCODECOMPASS_LINKER=gold -DCMAKE_BUILD_TYPE=Release ../Source
-- The C compiler identification is GNU 7.5.0
-- The CXX compiler identification is GNU 7.5.0
[...]
-- Found Threads: TRUE
-- Found Boost: /usr/include (found version "1.65.1") found components:  filesystem log program_options regex system thread date_time log_setup chrono atomic
-- Found Java: /usr/bin/java (found version "11.0.7")
-- Found ODB: ~/Work/CodeCompass/Dependencies-Install/bin/odb  found components:  sqlite missing components:  pgsql
-- Found Thrift version 0.13.0: ~/Work/CodeCompass/Dependencies-Install/bin/thrift
-- Found GTest: ~/Work/CodeCompass/Dependencies-Install/lib/libgtest.a
-- Found the following CodeCompass plugins:
--   cpp
--   cpp_reparse
--   dummy
--   git
--   metrics
--   search
Loading plugin cpp
-- Found LLVM 7.0.0
-- Using LLVMConfig.cmake in: /usr/lib/llvm-7/cmake
-- Using ClangConfig.cmake in: /usr/lib/cmake/clang-7
-- Using RTTI for LLVM: ON
Skipping generation of test project cpptest.
Loading plugin cpp_reparse
Loading plugin dummy
Loading plugin git
-- Found libgit2: /usr/lib/x86_64-linux-gnu/libgit2.so
Loading plugin metrics
Loading plugin search
-- Configuring done
-- Generating done
-- Build files have been written to: ~/Work/CodeCompass/filbetest

@whisperity
Copy link
Contributor

All my recent patches were based upon this branch: https://github.com/whisperity/CodeCompass/commits/working-master. There is only one commit, e0190b6, on top of you adding a few includes here and there in this patch, which allowed me to get master working again.

Upgraded the script, not to leave any garbage if it fails to run
properly.
@mcserep
Copy link
Collaborator

mcserep commented May 28, 2020

@whisperity Guess I have managed to solve it. Here is the case:

  • Until now FindOdb.cmake was configured to try to find both sqlite and pgsql components; but somehow (not clear right now for me how) it only linked against the proper library.
  • Now with ODB 2.5.0 beta it links against both libraries which for some another reason messes up the cpp_reparse plugin (which I did not analyze how it works).

So in 57ba7bb I modified FindOdb.cmake to only find the ODB component which is set in the DATABASE variable. As a result only the needed component will be found and linked, and cpp_reparse does not segfault on webserver startup.

@mcserep
Copy link
Collaborator

mcserep commented May 28, 2020

From me it's an approve on this PR, thanks for your work @filbeofITK . Please write if you have any major concerns @whisperity , @csordasmarton and @bruntib and let's get this PR merged ASAP, so the master branch could be compiled again on Bionic.
Since the issue is critical, I advise that minor concerns shall go to a separate PR.

Copy link
Contributor

@whisperity whisperity left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a few problems with the install guide that makes it not work on a reasonably fresh system. I've done the installation from scratch on an Ubuntu 18.04.4 LTS installed this morning into an emulator.
I'm attaching suggestions everywhere as I figured out what to do to make it build!

You read it right: the branch works and builds, with the suggestions (usually a missing package here and there on the system!) applied.

doc/deps.md Outdated
@@ -106,21 +158,13 @@ cd thrift-0.13.0

# Ant is required for having Java support in Thrift.
sudo apt-get install ant
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe this is for me, but simply running the build on 18.04 does not work in 2020. I'm testing the build on a brand new virtual machine installed this morning.
Even though Gradle started not to support HTTP download in 2020, Thrift 0.13 -- released 2019 Oct -- is still using an HTTP URL. The following output happens during the build, towards the end:

make[3]: Entering directory '/home/codecompass/thrift-0.13.0/lib/java'
./gradlew  assemble \
        -Prelease=true \
        -Pthrift.version=0.13.0 \
        --console=plain
> Task :copyDependencies FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':copyDependencies'.
> Could not resolve all files for configuration ':testRuntime'.
   > Could not resolve org.slf4j:slf4j-api:1.7.25.
     Required by:
         project :
      > Could not resolve org.slf4j:slf4j-api:1.7.25.
         > Could not get resource 'http://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.7.25/slf4j-api-1.7.25.pom'.
            > Could not GET 'http://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.7.25/slf4j-api-1.7.25.pom'. Received status code 501 from server: HTTPS Required
   > Could not resolve org.apache.httpcomponents:httpclient:4.5.6.
     Required by:
         project :
      > Could not resolve org.apache.httpcomponents:httpclient:4.5.6.
         > Could not get resource 'http://repo1.maven.org/maven2/org/apache/httpcomponents/httpclient/4.5.6/httpclient-4.5.6.pom'.
            > Could not GET 'http://repo1.maven.org/maven2/org/apache/httpcomponents/httpclient/4.5.6/httpclient-4.5.6.pom'. Received status code 501 from server: HTTPS Required
   > Could not resolve org.apache.httpcomponents:httpcore:4.4.1.

The relevant breaking line is in lib/java/gradle.properties:

mvn.repo=http://repo1.maven.org/maven2

Swapping HTTP to HTTPS in the Gradle cfg fixes this:

Suggested change
sudo apt-get install ant
sudo apt-get install ant
# Fix Thrift install script not using HTTPS for Java dependencies (http:// -> https://)
sed 's/http:\/\//https:\/\//g' -i ./lib/java/gradle.properties

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will check this in a Docker container on a fresh system soon.

Also, we are at this point of the guide, we don't need Ant to be installed. I don't have ant installed. The Docker environment does not install ant.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My conclusion is that for CodeCompass we only need the Java generator and not the Java library, which we have also discussed through emails last November.

Without Ant the Java library won't be built and there is no HTTP->HTTPS issue with Gradle. (We could even add the --without-java flag to Thrift, actually the CI was building CodeCompass that way for quite a long time previously.)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mcserep Did you check this with running a server and trying out the search parser and search service? Those are our only upstream stuff that "depend on" Java, but I am not sure about their inside architecture.

@whisperity
Copy link
Contributor

@whisperity Guess I have managed to solve it. Here is the case:

* Until now `FindOdb.cmake` was configured to try to find both _sqlite_ and _pgsql_ components; but somehow (not clear right now for me how) it only linked against the proper library.

* Now with ODB 2.5.0 beta it links against **both** libraries which for some another reason messes up the `cpp_reparse` plugin (which I did not analyze how it works).

@mcserep Right, thanks! I was thinking it's some weird link-level ODR yadda... Anyways, we shan't depend on the database that is not to be used by the build, so this clears it up.

@whisperity
Copy link
Contributor

🎉 And happy news: the built binaries also work. I have parsed the built CodeCompass, with all navigation, reparse, search, etc.

Screenshot_20200529_123632
Screenshot_20200529_123422

@whisperity
Copy link
Contributor

@mcserep Please check my suggestions above, especially about the system packages. There is a dissonance between the Travis- and the Dockerfile vs. going on the guide manually which is not user friendly. Otherwise, thank you for all the work coordinating this patch! 🏆

@filbeofITK
Copy link
Contributor Author

Thanks @whisperity for this detailed review. I have one question about the ODB versioning. Even thought ODB 2.4 works on Ubuntu 16.04 just fine I believe we should use the newer version there too, because of consistency. That's why I changed the travis build like I did at first. And that caused the inconsistency that you mentioned.

@whisperity
Copy link
Contributor

@filbeofITK I'm not sure. I'd vote for not resorting to self-compile on a system where it works. Ubuntu 16.04 will have it's support terminated in less than a year... Until we do not depend on anything from ODB 2.5.0 that is not available in 2.4.0, let's not require users to do extra steps.

Please can you squash the commits and tidy the patch out for merging? It has become 25 commits by now, most of which are few-liners.

@mcserep
Copy link
Collaborator

mcserep commented May 29, 2020

Please can you squash the commits and tidy the patch out for merging? It has become 25 commits by now, most of which are few-liners.

I think we can depend on the "Squash and merge" feature of GitHub. The one who merges shall select that button and everything will be squashed properly.

Co-authored-by: Whisperity <whisperity@gmail.com>
Co-authored-by: Whisperity <whisperity@gmail.com>
@mcserep
Copy link
Collaborator

mcserep commented May 29, 2020

@mcserep Please check my suggestions above, especially about the system packages. There is a dissonance between the Travis- and the Dockerfile vs. going on the guide manually which is not user friendly. Otherwise, thank you for all the work coordinating this patch!

Thanks for the through review of the PR and the install guide @whisperity !

I think the ODB version question on Ubuntu 16.04 is though, as both options have advantages and backwards. While it is true that uing ODB 2.5-beta eliminates any version difference issue between the 2 supported Ubuntu version, it also adds the additional complexity of compiling ODB, when it is not really necessary. If the developers behind ODB are nice guys and take semantic versioning seriously, there should be no API breaking change between ODB 2.4 and 2.5.

Since package-repository installed dependencies may already diverge between Xenial and Bionic, I guess it is not so bad if the ODB version differs too. So I agree with @whisperity in this topic.

Co-authored-by: Whisperity <whisperity@gmail.com>
@mcserep mcserep merged commit 6c27226 into Ericsson:master May 29, 2020
bruntib pushed a commit to bruntib/CodeCompass that referenced this pull request Jun 20, 2020
Switched to odb 2.5 from 2.4, since the version 2.4 is currently broken.

Co-authored-by: Bendegúz Filyó <filyo.bendeguz@hallgato.ppke.hu>
Co-authored-by: Whisperity <whisperity@gmail.com>
Co-authored-by: Máté Cserép <mcserep@gmail.com>
@filbeofITK filbeofITK deleted the odb_upgrade branch September 7, 2020 09:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Kind: Critical 💥 Target: Developer environment Developer environment issues consist of CodeCompass or 3rd-party build tooling, configuration or CI. Target: Documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ODB error with gcc-7.5.0

5 participants