Skip to content

Commit 6481b34

Browse files
alessandrogarioSmjerttheopolis
authored
Refactor third-party libraries to build from source on Linux (osquery#5706)
Add a way to compile third-party libraries from source instead of downloading prebuilt ones. Each library source code is downloaded with git into a submodule at configure time, in response to the find_package(library_name) CMake call, except for OpenSSL where the official source archive is used. Each submodule is attached to a release tag on its own upstream repository. All the libraries are built using CMake directly, except for OpenSSL which uses a formula system, which permits to build libraries with a separate build system when there's no easy way to integrate it directly with CMake. This new dependency system determines which library is fetched from where using the concept of "layers". Currently we have three of them: source, formula, facebook, where the last layer represents the pre-built libraries. The provided order will be used when looking for libraries. A system to patch submodule source code has been added and it's currently used with googletest, libudev and util-linux. Patches should be put under libraries/cmake/source/<library name>/patches/<submodule>, where <submodule> is often one and is "src", but in other cases, like AWS, there are multiple with a more specific name. If for whatever reason the submodule cloning or the patching fails, the submodule has to be unregistered and its folder should be cleared. This should be achievable with "git submodule deinit -f <submodule path>" Following some other changes on existing functionality: - Changed the CMake variable BUILD_TESTING to OSQUERY_BUILD_TESTS to avoid enabling tests on third party libraries. Due to an issue with glog the BUILD_TESTING variable will be always forced to OFF. - Moved compiler and linker flags to their own file cmake/flags.cmake - Moved all the third-party CMakeLists.txt used for pre-built libraries under libraries/cmake/facebook - Added the --exclude-folders option to tools/format-check.py and tools/git-clang-format.py, so that it's possible to ignore any third party library source code. - The format and format_check target use the new --exclude-folders option to exclude libraries/cmake/source from formatting. - The test and osquery binaries are properly compiled with PIE (osquery#5611) Co-authored-by: Stefano Bonicatti <[email protected]> Co-authored-by: Teddy Reed <[email protected]>
1 parent ecf9856 commit 6481b34

File tree

456 files changed

+50861
-635
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

456 files changed

+50861
-635
lines changed

‎.gitignore

-1
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,6 @@ build/
3030
# Run Artifacts
3131
*.log
3232
*.orig
33-
*.patch
3433
*.rej
3534

3635
# Vagrant Artifacts

‎.gitmodules

+126
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
[submodule "libraries/cmake/source/bzip2/src"]
2+
path = libraries/cmake/source/bzip2/src
3+
url = https://sourceware.org/git/bzip2.git
4+
[submodule "libraries/cmake/source/libarchive/src"]
5+
path = libraries/cmake/source/libarchive/src
6+
url = https://github.com/libarchive/libarchive
7+
[submodule "libraries/cmake/source/libmagic/src"]
8+
path = libraries/cmake/source/libmagic/src
9+
url = https://github.com/file/file
10+
[submodule "libraries/cmake/source/libudev/src"]
11+
path = libraries/cmake/source/libudev/src
12+
url = https://github.com/systemd/systemd
13+
[submodule "libraries/cmake/source/libxml2/src"]
14+
path = libraries/cmake/source/libxml2/src
15+
url = https://github.com/GNOME/libxml2
16+
[submodule "libraries/cmake/source/linenoise-ng/src"]
17+
path = libraries/cmake/source/linenoise-ng/src
18+
url = https://github.com/arangodb/linenoise-ng
19+
[submodule "libraries/cmake/source/lzma/src"]
20+
path = libraries/cmake/source/lzma/src
21+
url = https://github.com/xz-mirror/xz
22+
[submodule "libraries/cmake/source/rapidjson/src"]
23+
path = libraries/cmake/source/rapidjson/src
24+
url = https://github.com/Tencent/rapidjson
25+
[submodule "libraries/cmake/source/rocksdb/src"]
26+
path = libraries/cmake/source/rocksdb/src
27+
url = https://github.com/facebook/rocksdb
28+
[submodule "libraries/cmake/source/sleuthkit/src"]
29+
path = libraries/cmake/source/sleuthkit/src
30+
url = https://github.com/sleuthkit/sleuthkit
31+
[submodule "libraries/cmake/source/sqlite/src"]
32+
path = libraries/cmake/source/sqlite/src
33+
url = https://github.com/osquery/third-party-sqlite
34+
[submodule "libraries/cmake/source/ssdeep-cpp/src"]
35+
path = libraries/cmake/source/ssdeep-cpp/src
36+
url = https://github.com/ssdeep-project/ssdeep
37+
[submodule "libraries/cmake/source/thrift/src"]
38+
path = libraries/cmake/source/thrift/src
39+
url = https://github.com/apache/thrift
40+
[submodule "libraries/cmake/source/util-linux/src"]
41+
path = libraries/cmake/source/util-linux/src
42+
url = https://git.kernel.org/pub/scm/utils/util-linux/util-linux
43+
[submodule "libraries/cmake/source/yara/src"]
44+
path = libraries/cmake/source/yara/src
45+
url = https://github.com/VirusTotal/yara
46+
[submodule "libraries/cmake/source/zlib/src"]
47+
path = libraries/cmake/source/zlib/src
48+
url = https://github.com/madler/zlib
49+
[submodule "libraries/cmake/source/zstd/src"]
50+
path = libraries/cmake/source/zstd/src
51+
url = https://github.com/facebook/zstd
52+
[submodule "libraries/cmake/source/augeas/src"]
53+
path = libraries/cmake/source/augeas/src
54+
url = https://github.com/hercules-team/augeas
55+
[submodule "libraries/cmake/source/smartmontools/src"]
56+
path = libraries/cmake/source/smartmontools/src
57+
url = https://github.com/osquery/third-party-smartmontools
58+
[submodule "libraries/cmake/source/berkeley-db/src"]
59+
path = libraries/cmake/source/berkeley-db/src
60+
url = https://github.com/osquery/third-party-berkeley-db
61+
[submodule "libraries/cmake/source/librpm/src"]
62+
path = libraries/cmake/source/librpm/src
63+
url = https://github.com/rpm-software-management/rpm
64+
[submodule "libraries/cmake/source/popt/src"]
65+
path = libraries/cmake/source/popt/src
66+
url = https://github.com/osquery/third-party-popt
67+
[submodule "libraries/cmake/source/libdpkg/src"]
68+
path = libraries/cmake/source/libdpkg/src
69+
url = https://git.dpkg.org/cgit/dpkg/dpkg.git
70+
[submodule "libraries/cmake/source/libaudit/src"]
71+
path = libraries/cmake/source/libaudit/src
72+
url = https://github.com/linux-audit/audit-userspace
73+
[submodule "libraries/cmake/source/libelfin/src"]
74+
path = libraries/cmake/source/libelfin/src
75+
url = https://github.com/aclements/libelfin
76+
[submodule "libraries/cmake/source/libgcrypt/src"]
77+
path = libraries/cmake/source/libgcrypt/src
78+
url = https://dev.gnupg.org/source/libgcrypt.git
79+
[submodule "libraries/cmake/source/libgpg-error/src"]
80+
path = libraries/cmake/source/libgpg-error/src
81+
url = https://dev.gnupg.org/source/libgpg-error.git
82+
[submodule "libraries/cmake/source/libcryptsetup/src"]
83+
path = libraries/cmake/source/libcryptsetup/src
84+
url = https://gitlab.com/cryptsetup/cryptsetup.git
85+
[submodule "libraries/cmake/source/libdevmapper/src"]
86+
path = libraries/cmake/source/libdevmapper/src
87+
url = https://github.com/lvmteam/lvm2
88+
[submodule "libraries/cmake/source/libiptables/src"]
89+
path = libraries/cmake/source/libiptables/src
90+
url = https://github.com/osquery/third-party-iptables
91+
[submodule "libraries/cmake/source/librdkafka/src"]
92+
path = libraries/cmake/source/librdkafka/src
93+
url = https://github.com/edenhill/librdkafka
94+
[submodule "libraries/cmake/source/lldpd/src"]
95+
path = libraries/cmake/source/lldpd/src
96+
url = https://github.com/vincentbernat/lldpd
97+
[submodule "libraries/cmake/source/googletest/src"]
98+
path = libraries/cmake/source/googletest/src
99+
url = https://github.com/google/googletest
100+
[submodule "libraries/cmake/source/glog/src"]
101+
path = libraries/cmake/source/glog/src
102+
url = https://github.com/google/glog
103+
[submodule "libraries/cmake/source/gflags/src"]
104+
path = libraries/cmake/source/gflags/src
105+
url = https://github.com/gflags/gflags
106+
[submodule "libraries/cmake/source/aws-sdk-cpp/aws-c-common_src"]
107+
path = libraries/cmake/source/aws-sdk-cpp/aws-c-common_src
108+
url = https://github.com/awslabs/aws-c-common
109+
[submodule "libraries/cmake/source/aws-sdk-cpp/aws-c-event-stream_src"]
110+
path = libraries/cmake/source/aws-sdk-cpp/aws-c-event-stream_src
111+
url = https://github.com/awslabs/aws-c-event-stream
112+
[submodule "libraries/cmake/source/aws-sdk-cpp/aws-checksums_src"]
113+
path = libraries/cmake/source/aws-sdk-cpp/aws-checksums_src
114+
url = https://github.com/awslabs/aws-checksums
115+
[submodule "libraries/cmake/source/aws-sdk-cpp/aws-sdk-cpp_src"]
116+
path = libraries/cmake/source/aws-sdk-cpp/aws-sdk-cpp_src
117+
url = https://github.com/aws/aws-sdk-cpp
118+
[submodule "libraries/cmake/source/boost/src"]
119+
path = libraries/cmake/source/boost/src
120+
url = https://github.com/boostorg/boost
121+
[submodule "libraries/cmake/source/icu/src"]
122+
path = libraries/cmake/source/icu/src
123+
url = https://github.com/unicode-org/icu
124+
[submodule "libraries/cmake/source/augeas/gnulib/src"]
125+
path = libraries/cmake/source/augeas/gnulib/src
126+
url = https://github.com/osquery/third-party-gnulib

‎CMakeLists.txt

+114-5
Original file line numberDiff line numberDiff line change
@@ -5,15 +5,19 @@
55
# the LICENSE file found in the root directory of this source tree.
66

77
cmake_minimum_required(VERSION 3.13.3)
8+
9+
cmake_policy(SET CMP0083 NEW)
10+
811
project(osquery)
912

10-
if(BUILD_TESTING)
13+
if(OSQUERY_BUILD_TESTS)
1114
enable_testing()
1215
endif()
1316

1417
include(cmake/globals.cmake)
1518
include(cmake/utilities.cmake)
1619
include(cmake/options.cmake)
20+
include(cmake/flags.cmake)
1721
include(cmake/packaging.cmake)
1822

1923
function(main)
@@ -32,25 +36,130 @@ function(main)
3236
endif()
3337
endif()
3438

35-
generateGlobalSettingsTargets()
36-
3739
findPythonExecutablePath()
3840
generateSpecialTargets()
3941

40-
add_subdirectory("third-party")
42+
add_subdirectory("libraries")
43+
importLibraries()
44+
4145
add_subdirectory("osquery")
4246
add_subdirectory("plugins")
4347
add_subdirectory("tools")
4448
add_subdirectory("specs")
4549

46-
if(BUILD_TESTING)
50+
if(OSQUERY_BUILD_TESTS)
4751
add_subdirectory("tests")
4852
endif()
4953

5054
identifyPackagingSystem()
5155
generateInstallTargets()
5256
generatePackageTarget()
57+
endfunction()
58+
59+
function(importLibraries)
60+
set(python_module_list
61+
markupsafe
62+
jinja2
63+
)
64+
65+
set(library_descriptor_list
66+
"Linux,Darwin:augeas"
67+
"Linux:berkeley-db"
68+
"Linux,Darwin,Windows:boost"
69+
"Linux,Darwin,Windows:bzip2"
70+
"Linux,Darwin,Windows:gflags"
71+
"Linux,Darwin:glibc"
72+
"Linux,Darwin,Windows:glog"
73+
"Linux,Darwin,Windows:googletest"
74+
"Linux,Darwin,Windows:libarchive"
75+
"Linux:libaudit"
76+
"Linux:libcryptsetup"
77+
"Linux:libdevmapper"
78+
"Linux:libdpkg"
79+
"Linux:libelfin"
80+
"Linux:libgcrypt"
81+
"Linux:libgpg-error"
82+
"Linux:libiptables"
83+
"Linux,Darwin:libmagic"
84+
"Linux,Darwin:librdkafka"
85+
"Linux:librpm"
86+
"Linux:libudev"
87+
"Linux,Darwin:libxml2"
88+
"Linux,Darwin,Windows:linenoise-ng"
89+
"Linux,Darwin:lldpd"
90+
"Linux:lzma"
91+
"Linux,Darwin:popt"
92+
"Linux,Darwin,Windows:rapidjson"
93+
"Linux,Darwin,Windows:rocksdb"
94+
"Linux,Darwin:sleuthkit"
95+
"Linux,Darwin:smartmontools"
96+
"Linux,Darwin,Windows:sqlite"
97+
"Linux,Darwin:ssdeep-cpp"
98+
"Linux,Darwin,Windows:thrift"
99+
"Linux:util-linux"
100+
"Linux,Darwin:yara"
101+
"Linux,Darwin,Windows:zlib"
102+
"Linux,Darwin,Windows:zstd"
103+
"Linux,Darwin,Windows:openssl"
104+
"Linux,Darwin,Windows:aws-sdk-cpp"
105+
)
106+
107+
foreach(python_module ${python_module_list})
108+
find_package("${python_module}" REQUIRED)
109+
endforeach()
110+
111+
foreach(library_descriptor ${library_descriptor_list})
112+
# Expand the library descriptor
113+
string(REPLACE ":" ";" library_descriptor "${library_descriptor}")
114+
115+
list(GET library_descriptor 0 platform_list)
116+
list(GET library_descriptor 1 library)
117+
118+
string(REPLACE "," ";" platform_list "${platform_list}")
53119

120+
list(FIND platform_list "${CMAKE_SYSTEM_NAME}" platform_index)
121+
if(platform_index EQUAL -1)
122+
continue()
123+
endif()
124+
125+
find_package("${library}" REQUIRED)
126+
127+
# Facebook-provided libraries always come with the thirdparty_ prefix
128+
if(TARGET "thirdparty_${library}")
129+
continue()
130+
131+
# For generic libraries that import the library name, let's create
132+
# an alias
133+
elseif(TARGET "${library}")
134+
add_library("thirdparty_${library}" ALIAS "${library}")
135+
136+
# Legacy libraries will just export variables; build a new INTERFACE
137+
# target with them
138+
elseif(DEFINED "${library}_LIBRARIES")
139+
if(NOT DEFINED "${library}_INCLUDE_DIRS")
140+
message(FATAL_ERROR "Variable ${library}_INCLUDE_DIRS was not found!")
141+
endif()
142+
143+
add_library("thirdparty_${library}" INTERFACE)
144+
145+
target_link_libraries("thirdparty_${library}" INTERFACE
146+
${library}_LIBRARIES
147+
)
148+
149+
target_include_directories("thirdparty_${library}" INTERFACE
150+
${library}_INCLUDE_DIRS
151+
)
152+
153+
if(DEFINED "${library}_DEFINITIONS")
154+
target_compile_definitions("thirdparty_${library}" INTERFACE
155+
${library}_DEFINITIONS
156+
)
157+
endif()
158+
159+
else()
160+
message(FATAL_ERROR "The '${library}' was found but it couldn't be imported correctly")
161+
endif()
162+
endforeach()
54163
endfunction()
55164

56165
main()

‎azure-pipelines.yml

+7-5
Original file line numberDiff line numberDiff line change
@@ -20,9 +20,11 @@ jobs:
2020
vmImage: 'Ubuntu-16.04'
2121

2222
container:
23-
image: trailofbits/osql-experimental:ubuntu-18.04
23+
image: trailofbits/osquery:ubuntu-18.04
2424
options: --privileged
2525

26+
timeoutInMinutes: 120
27+
2628
steps:
2729
- script: mkdir $(Build.BinariesDirectory)/build
2830
displayName: "Create build folder"
@@ -35,7 +37,7 @@ jobs:
3537
-DCMAKE_BUILD_TYPE=$(BUILD_TYPE)
3638
-DCMAKE_C_COMPILER=clang
3739
-DCMAKE_CXX_COMPILER=clang++
38-
-DBUILD_TESTING=ON
40+
-DOSQUERY_BUILD_TESTS=ON
3941
$(EXTRA_CMAKE_ARGS)
4042
$(Build.SourcesDirectory)
4143

@@ -141,7 +143,7 @@ jobs:
141143
steps:
142144
- script: |
143145
brew upgrade
144-
brew install ccache
146+
brew install ccache flex bison
145147
displayName: "Install Homebrew and prerequisites"
146148
timeoutInMinutes: 20
147149
@@ -152,7 +154,7 @@ jobs:
152154
displayName: "Configure osquery"
153155
inputs:
154156
workingDirectory: $(Build.BinariesDirectory)/build
155-
cmakeArgs: -DCMAKE_BUILD_TYPE=$(BUILD_TYPE) -DBUILD_TESTING=ON $(EXTRA_CMAKE_ARGS) $(Build.SourcesDirectory)
157+
cmakeArgs: -DCMAKE_BUILD_TYPE=$(BUILD_TYPE) -DOSQUERY_BUILD_TESTS=ON $(EXTRA_CMAKE_ARGS) $(Build.SourcesDirectory)
156158

157159
- task: CMake@1
158160
displayName: "Build osquery"
@@ -251,7 +253,7 @@ jobs:
251253
displayName: "Configure osquery"
252254
inputs:
253255
workingDirectory: $(Build.BinariesDirectory)\build
254-
cmakeArgs: -G "Visual Studio 15 2017 Win64" -T host=x64 -DBUILD_TESTING=ON $(Build.SourcesDirectory)
256+
cmakeArgs: -G "Visual Studio 15 2017 Win64" -T host=x64 -DOSQUERY_BUILD_TESTS=ON $(Build.SourcesDirectory)
255257

256258
- task: CMake@1
257259
displayName: "Build osquery"

0 commit comments

Comments
 (0)