From: Cyril Brulebois Date: Wed, 31 May 2023 16:54:17 +0000 (+0100) Subject: Import crowdsec_1.4.6-4.debian.tar.xz X-Git-Tag: archive/raspbian/1.4.6-4+rpi1^2~13^4 X-Git-Url: https://dgit.raspbian.org/?a=commitdiff_plain;h=4d7371513ff9f0c8e00fe0fb8ea0454757290605;p=crowdsec.git Import crowdsec_1.4.6-4.debian.tar.xz [dgit import tarball crowdsec 1.4.6-4 crowdsec_1.4.6-4.debian.tar.xz] --- 4d7371513ff9f0c8e00fe0fb8ea0454757290605 diff --git a/README.Debian b/README.Debian new file mode 100644 index 0000000..66331e5 --- /dev/null +++ b/README.Debian @@ -0,0 +1,126 @@ +CrowdSec for Debian +=================== + +# Local API and Central API + +There are multiple ways to configure `crowdsec`, leveraging a Local +API (LAPI) and/or the Central API (CAPI). + + +At the moment, the default configuration does the following: + + 1. Set up a Local API locally, that doesn't listen on the + network. This can be adjusted by following the + [upstream local API documentation](https://doc.crowdsec.net/docs/local_api/intro). + + 1. Register to the Central API by default, to take part in the + collective effort. If that's not desired, it is possible to create + a `/etc/crowdsec/online_api_credentials.yaml` file before + installing the package, that contains a comment (e.g. + `# no thanks`). In this case, the registration is skipped, and + this file is also left behind in case the package is purged, so as + to respective the admin's wishes if later reinstalled. If one + reconsiders, it's sufficient to empty this file and run the + following command manually: + + cscli capi register + + +# Hub management + +## Offline hub + +The `crowdsec` Debian package ships a copy of the available +collections (parsers, scenarios, and some other items) on the online +[hub](https://hub.crowdsec.net/) so that it can be configured out of +the box, without having to download anything from the internet. For +the purpose of this document, let's call this copy the “offline hub”. + +Starting with the 1.4.6 release, only 3 collections (linux, apache2, and +nginx) are enabled automatically, as recommended by upstream. When +upgrading from earlier versions that enabled everything automatically, +*if* those 3 collections are still enabled, the postinst script makes +sure their dependencies are enabled as well. Otherwise, collections are +left untouched. + +Enabling collections is achieved by creating symlinks below the +`/etc/crowdsec` directories, for collections, parsers, postoverflows, +and scenarios. + + +## Online hub + +It is also possible to move away from the local, offline hub to the +online hub, so as to benefit from new or updated items without having +to wait for a package update. To do so, follow the upstream docs and +run: + + cscli hub update + +Once that has happened, the offline hub will no longer be considered +and only items from the online hub will be used. + +If going back to the offline hub is desired, that can be achieved by +running this command: + + /var/lib/dpkg/info/crowdsec.postinst disable-online-hub + +It will undo the previous `enable-online-hub` action that happened +automatically when calling `cscli hub update` the first time, +but it might remove items that were available on the online hub, but +that are not on the offline hub. One might want to double check the +state of all configured collections afterward. + +Once that has happened, don't forget to restart the crowdsec unit: + + systemctl restart crowdsec.service + + +## Implementation details + +When configuring a collection, symlinks are created under +`/etc/crowdsec`, pointing at items under `/var/lib/crowdsec/hub`. + +Initially, that directory points at items from the offline hub, +shipped under `/usr/share/crowdsec/hub`. + +When switching to the online hub, the `/var/lib/crowdsec/hub` +directory no longer points at the offline hub, and contains a copy of +items downloaded from instead. + +If switching back to the offline hub, `/var/lib/crowdsec/hub` is +cleaned up (downloaded items are removed), and it starts pointing at +the offline hub again. + + +# Configuration management + +The `crowdsec` package ships a few files under `/etc/crowdsec`, registered via +`conffiles`: + + - `/etc/crowdsec/acquis.yaml` + - `/etc/crowdsec/config.yaml` + - `/etc/crowdsec/patterns/*` + - `/etc/crowdsec/profiles.yaml` + - `/etc/crowdsec/simulation.yaml` + +Starting with version 1.4.2, it's possible to add or override settings by using +a `.local` suffix for some of those files, as +[documented upstream](https://docs.crowdsec.net/docs/configuration/crowdsec_configuration/#configuration-files-that-support-yamllocal). + +It's probably a good idea to use this mechanism so that further `crowsec` +updates don't generate too many `dpkg` prompts about configuration files that +have been modified locally. + +Starting with version 1.4.2 as well, the `postinst` script performs some +detection regarding the database. If it's using the `sqlite` driver (the +default) and if it doesn't look like that file is stored over NFS, then a +`config.yaml.local` file is generated automatically, enabling the [WAL +optimization](https://www.sqlite.org/wal.html). + +Upstream's default is an unconfigured +[`use_wal` setting](https://docs.crowdsec.net/docs/next/configuration/crowdsec_configuration/#use_wal) +which leads to warnings for everyone, so it was agreed with upstream to have the +Debian package try and guess the right value instead of leaving that up to each +and every `crowdsec` admin, and to store the result in the `config.yaml.local` +override. diff --git a/README.source b/README.source new file mode 100644 index 0000000..c7cbc2c --- /dev/null +++ b/README.source @@ -0,0 +1,91 @@ +Source instructions +=================== + +# Introduction + +The main orig tarball for this package matches upstream `crowdsec` releases, and +can be downloaded via `uscan`. Upstream code includes a `debian/` directory +(used for their pragmatic packages), that gets overwritten during merges. + +In addition to that main tarball, two additional tarballs are used: + + - one for extra data; + - one for the offline hub. + +The idea behind those tarballs is making `crowdsec` useful out of the box, +without having to download anything from the internet (which it can and would do +on its own if that approach hadn't been implemented). + + +# Tarball creation + +## `data` tarball + +The `data` tarball ships files that are found in various other locations: + + - text and regex files found in https://github.com/crowdsecurity/sec-lists/ + - IPs found at https://www.cloudflare.com/ips-v4 + - IPs found at https://www.cloudflare.com/ips-v6 + +Since those updates are infrequent, the tarball must be crafted manually. It can +be carried over from the previous `crowdsec` upstream release if there were no +updates; in that case, it's only a matter of adjusting the version number in the +filename. + + +## `hub` tarball + +The `hub` tarball ships files found in the +[crowdsecurity/hub](https://github.com/crowdsecurity/hub) repository. It can be +built from the `master` when working on release candidates, or from the branch +matching the upstream release once the final release is out (e.g. `v1.4.2`). + +There are a few variables and rules in `debian/rules` to automate generating +this tarball as this is much more frequent than updating the `data` tarball: + + - `HUB_ID` is an integer that is used both in the tarball's filename and in the + hub directory name where its contents are extracted. Use `1` when packaging a + new upstream release of `crowdsec`, and increment it if the hub needs to be + updated while `crowdsec` itself stays at the same version. As of December + 2022, there has been no need for such increments, so documentation and/or + code might be lacking. + - `HUB_BRANCH` is the name of the branch to generate the export from. As + mentioned above, `origin/master` is to be used for `crowdsec` release + candidates, and `origin/vX.Y.Z` can be used instead once the `crowdsec` final + release is out and the matching branch is available in the `hub` repository. + - `HUB_DIR` is an absolute or relative path to the `hub` checkout. That + checkout only needs to be kept current (e.g. via `git fetch`), no local + branches are needed if the `origin/` prefix is specified in `HUB_BRANCH`. + +Once all those variables have been set appropriately, the tarball can be +generated with: + + ./debian/rules generate_hub_tarball + +It's best practice to include the output in a `debian/changelog` entry, to keep +track of the upstream commit that was used to generate that export. + + +Example: + + $ ./debian/rules generate_hub_tarball + […] + Generated hub tarball from branch origin/v1.4.2, at commit 73d2edaaf1 + +can be documented this way: + + * Include a snapshot of hub files from the v1.4.2 branch, at commit + 73d2edaaf1. + + +# Tarball usage + +Given a clean git repository, the following commands can be used to extract data +and hub tarballs into place, creating `dataX/` and `hubY/` directories: + + ./debian/rules extract_data_tarball + ./debian/rules extract_hub_tarball + +It's important to have them in place before trying to apply patches, as some of +them modify hub files; that's needed at least to disable `geoip-enrich` since +the MMDB files can't be distributed. diff --git a/changelog b/changelog new file mode 100644 index 0000000..7a58f84 --- /dev/null +++ b/changelog @@ -0,0 +1,362 @@ +crowdsec (1.4.6-4) unstable; urgency=medium + + * Implement support for pending registration: since bouncers list crowdsec + in Recommends, we cannot guarantee the order in which bouncers and + crowdsec are configured (See: #1035499, #1036985). Bouncers can now + queue triplets (systemd unit name, bouncer identifier and API key) in + /var/lib/crowdsec/pending-registration. crowdsec.postinst will register + those bouncers, and start their systemd units after removing that file + (satisfying their ConditionPathExists=! on it). + * Replace `exit 0` with `break` in the preceding code block. + + -- Cyril Brulebois Wed, 31 May 2023 18:54:17 +0200 + +crowdsec (1.4.6-3) unstable; urgency=medium + + * When performing an upgrade from pre-1.4.x versions, apply a workaround + to avoid losing CAPI decisions for several hours (Closes: #1033138): + delete alert(s) matching “Community blocklist”, and if at least one + deletion occurred, restart the daemon to force an immediate pull. + * Hardcode libsqlite3-0 (>= 3.35.0) in Depends to ensure Ent-generated + SQLite queries are understood (Closes: #1033132): otherwise, we would + get a dependency on libsqlite3-0 (>= 3.12.0) via shlibs, which is + clearly not enough. + * Backport upstream patch to fix building in the past/in the future (as + seen with reproducible builds), no longer hardcoding the expected year + for yearless timestamps: + - 0016-try-to-make-reproducible-build-work-2119.patch + + -- Cyril Brulebois Sun, 19 Mar 2023 00:25:07 +0100 + +crowdsec (1.4.6-2) unstable; urgency=medium + + * Prefix package version with `v` when setting BUILD_VERSION, e.g. + crowdsec/v1.4.6-2-linux-debian (Closes: #1031324). + * Stop shipping a logrotate configuration snippet, as crowdsec rotates + logs on its own via lumberjack.Logger, and that can be configured in + the main configuration file (/etc/crowdsec/config.yaml): + - Delete debian/crowdsec.logrotate + - Add debian/crowdsec.conffiles, marking /etc/logrotate.d/crowdsec + with the remove-on-upgrade flag. + * When purging the package, remove internally-rotated log files, in + addition to removing main log files and the logrotate-generated ones. + * Really enable upstream-recommended collections, not just the items + they require: everything would work fine without this, but those + collections wouldn't appear in `cscli collections list`, and they + wouldn't be upgraded if admins decided to switch to the online hub. + * Compensate for the missing collections when upgrading from 1.4.6-1 + specifically (including binNMUs). + * Enable the crowdsecurity/whitelists parser as well. + * When performing a fresh install (as determined by the postinst's being + called with just `configure`), and when /run/systemd/system exists, + apply a workaround for upstream issue #2120 based on crowdsec.log: + - Exit immediately if the unit is not active (e.g. the admin masked + it before installing the package). + - Check whether entries are received from the Central API, and exit + if that's the case. + - Check whether `received 0 new entries` is logged, and restart then + exit if that's the case. + - Repeat those checks every second, up to 20 times. + * When performing an upgrade from pre-1.4.x versions, apply a workaround + to avoid huge delays at the `restart` step: lower TimeoutStopSec from + 90s (default) to only 20s, using a runtime override (Closes: #1031326). + * Add a dh_install-indep override to get rid of files left over after + running the test suite (Closes: #1031328). + + -- Cyril Brulebois Fri, 17 Mar 2023 14:42:04 +0100 + +crowdsec (1.4.6-1) unstable; urgency=medium + + * New upstream release (Closes: #1031322). + * Include a snapshot of hub files from the v1.4.6 branch, at commit + f23a543a80. + * Delete patch: + - 0012-work-around-buggy-testparse-test.patch (fixed upstream) + * Extend patch to avoid crowdsecurity/linux's being marked tainted: + - 0004-disable-geoip-enrich.patch + * Extend patch to skip more unreliable tests: + - 0013-skip-flakky-tests.patch + * Add patches: + - 0014-silence-yaml-patching.patch: avoid polluting cscli's output + with debug messages. + - 0015-silence-not-latest-version.patch: upstream maintains a hub + branch for our stable release (Closes: #1031323). + * Rework collections handling: + - With crowdsec growing over time, the initial “let's enable all + collections” approach doesn't seem appropriate anymore. + - On initial installation, only enable 3 collections (and their + dependencies), which should cover common needs already: + + crowdsecurity/linux + + crowdsecurity/apache2 + + crowdsecurity/nginx + - On upgrade, check whether all 3 collections are (still) enabled. + If that's the case, enable their dependencies as well (as new + versions tend to gain dependencies over time). + - Let admins enable/disable any other collections on their own. + * Update README.Debian accordingly. + + -- Cyril Brulebois Thu, 02 Mar 2023 05:07:10 +0000 + +crowdsec (1.4.2-3) unstable; urgency=medium + + * Really fix FTBFS with -A, by really adjusting the override. + * Add patch: + - 0013-skip-flakky-tests.patch: this should avoid build failures, and + probably autopkgtest failures on “slow systems” (arm*, mips*). + + -- Cyril Brulebois Wed, 15 Feb 2023 10:03:10 +0000 + +crowdsec (1.4.2-2) unstable; urgency=medium + + * Fix FTBFS with -A (`Architecture: all` only) by adjusting the contents + of the crowdsec binary package within override_dh_install-arch instead + of override_dh_install. + + -- Cyril Brulebois Tue, 14 Feb 2023 23:54:17 +0000 + +crowdsec (1.4.2-1) unstable; urgency=medium + + * New upstream release (Closes: #1011665). + * Add patches: + - 0008-r3labs-diff-versions.patch: adjust import path for + r3labs/diff. + - 0009-disable-kafka-acquisition-module.patch: this would require + introducing a number of new packages. + - 0010-disable-some-tests.patch: cloudwatch, kinesis, and docker are + fine for integration tests but would require new packages and + setting up a test infrastructure. + - 0011-refresh-protobuf-code.patch: this avoids protobuf version + mismatch. + - 0012-work-around-buggy-testparse-test.patch: bump expected year + when parsing year-less timestamps. + * Refresh patches: + - 0003-adjust-systemd-unit.patch + - 0004-disable-geoip-enrich.patch + - 0005-adjust-config.patch + - 0007-automatically-enable-online-hub.patch + * Delete obsolete patches: + - 0001-use-a-local-machineid-implementation.patch (a separate package + is available now) + - 0002-add-compatibility-for-older-sqlite-driver.patch + - 0006-prefer-systemctl-restart.patch (bug fixed upstream) + - 0008-hub-disable-broken-scenario.patch + - 0009-Improve-http-bad-user-agent-use-regexp-197.patch + - 0010-5ae69aa293-fix-stacktrace-when-mmdb-files-are-not-present.patch + - 0011-4dbbd4b3c4-automatically-download-files-when-needed.patch + * Update dependencies: + - Add golang-entgo-ent-dev + - Add golang-github-alexliesenfeld-health-dev + - Add golang-github-aquasecurity-table-dev + - Add golang-github-beevik-etree-dev + - Add golang-github-blackfireio-osinfo-dev + - Add golang-github-c-robinson-iplib-dev + - Add golang-github-confluentinc-bincover-dev + - Add golang-github-crowdsecurity-dlog-dev + - Add golang-github-crowdsecurity-grokky-dev + - Add golang-github-crowdsecurity-machineid-dev + - Add golang-github-hashicorp-go-plugin-dev + - Add golang-github-ivanpirog-coloredcobra-dev + - Add golang-github-jackc-pgx-v4-dev + - Add golang-github-jarcoal-httpmock-dev + - Add golang-github-jszwec-csvutil-dev + - Add golang-github-masterminds-sprig-dev + - Add golang-github-pbnjay-memory-dev + - Add golang-github-r3labs-diff-dev + - Add golang-github-slack-go-slack-dev + - Add golang-github-texttheater-golang-levenshtein-dev + - Add golang-github-xhit-go-simple-mail-dev + - Bump golang-github-gin-gonic-gin-dev from 1.6.3 to 1.8.1 + - Delete golang-github-facebook-ent-dev (replaced with + golang-entgo-ent-dev) + - Delete golang-github-logrusorgru-grokky-dev (replaced with + golang-github-crowdsecurity-grokky-dev) + - Delete golang-github-olekukonko-tablewriter-dev (no longer used) + - Replace golang-logrus-dev with golang-github-sirupsen-logrus-dev + - Replace golang-pq-dev with golang-github-lib-pq-dev + - Replace golang-prometheus-client-dev with + golang-github-prometheus-client-golang-dev + - Replace golang-yaml.v2-dev with golang-gopkg-yaml.v2-dev + - Add python3 (for the testsuite). + * Adjust variables passed to the build system via LD_FLAGS: + - Set BUILD_CODENAME from platform/unix_common.mk + - Set BUILD_TIMESTAMP, replacing BUILD_DATE. + - Stop setting BUILD_GOVERSION, now determined at runtime. + * Avoid “date: invalid date ‘@’” warnings when using ancillary rules. + * Include a snapshot of hub files from the v1.4.2 branch, at commit + 73d2edaaf1. + * Add README.source, documenting additional tarballs. + * Include a snapshot of data files, at commit 03c7a30d70, plus + Cloudflare IPv4 and IPv6 addresses as of 2022-12-15. + * Upon upgrade, only enable items that are new since 1.0.9, instead + of enabling all of them (which could enable again things that were + disabled by the admin). + * Adjust lintian overrides: + - Adjust syntax to avoid mismatched overrides. + - Extend hardening-no-pie to plugins. + - Avoid a false positive with shell-script-fails-syntax-check. + - Avoid a false positive with broken-gz. + - Silence script-not-executable and unusual-interpreter bats + for *.bats file (test files). + + -- Cyril Brulebois Tue, 14 Feb 2023 22:32:27 +0000 + +crowdsec (1.0.9-4) experimental; urgency=medium + + * Add square brackets around paths for hardening-no-pie lintian overrides. + * Add library package, which is a build dependency bouncers require. + * Adjust debhelper control files for the switch from building a single + package to building multiple packages (mostly marking existing files + with a crowdsec. prefix). + * Adjust setting BUILD_CODENAME, avoiding an awk(ward) warning. + + -- Cyril Brulebois Fri, 06 May 2022 04:48:13 +0000 + +crowdsec (1.0.9-3) unstable; urgency=medium + + * Backport upstream patches to deal with missing MMDB files gracefully + (geolocation files aren't shipped by default): + - 5ae69aa293: fix stacktrace when mmdb files are not present (#935) + - 4dbbd4b3c4: automatically download files when needed (#895), so + that switching to the online hub doesn't require extra steps to + fetch files. + + -- Cyril Brulebois Sat, 04 Dec 2021 05:03:33 +0100 + +crowdsec (1.0.9-2) unstable; urgency=medium + + * Backport hub patch from upstream to fix false positives due to + substring matches (https://github.com/crowdsecurity/hub/pull/197): + + 0009-Improve-http-bad-user-agent-use-regexp-197.patch + + -- Cyril Brulebois Mon, 03 May 2021 07:29:06 +0000 + +crowdsec (1.0.9-1) unstable; urgency=medium + + * New upstream stable release: + + Improve documentation. + + Fix disabled Central API use case: without Central API credentials + in the relevant config file, crowdsec would still try and establish + a connection. + * Add patch to disable broken scenario (ban-report-ssh_bf_report, #181): + + 0008-hub-disable-broken-scenario.patch + * Add logrotate config for /var/log/crowdsec{,_api}.log (weekly, 4). + + -- Cyril Brulebois Mon, 15 Mar 2021 01:19:43 +0100 + +crowdsec (1.0.8-2) unstable; urgency=medium + + * Update postinst to also strip ltsich/ when installing symlinks + initially (new vendor in recent hub files, in addition to the usual + crowdsecurity/). + + -- Cyril Brulebois Tue, 02 Mar 2021 01:29:29 +0000 + +crowdsec (1.0.8-1) unstable; urgency=medium + + * New upstream stable release. + * Refresh patches: + + 0001-use-a-local-machineid-implementation.patch (unfuzzy) + + 0002-add-compatibility-for-older-sqlite-driver.patch + * Set cwversion variables through debian/rules (build metadata). + * Add patch so that upstream's crowdsec.service is correct on Debian: + + 0003-adjust-systemd-unit.patch + * Really add lintian overrides for hardening-no-pie warnings. + * Ship patterns below /etc/crowdsec/patterns: they're supposed to be + stable over time, and it's advised not to modify them, but let's allow + for some configurability. + * Include a snapshot of hub files from the master branch, at commit + d8a8509bdf: hub1. Further updates for a given crowdsec upstream + version will be numbered hubN. After a while, they will be generated + from a dedicated vX.Y.Z branch instead of from master. + * Implement a generate_hub_tarball target in debian/rules to automate + generating a tarball for hub files. + * Add patch to disable geoip-enrich in the hub files as it requires + downloading some files from the network that aren't under the usual + MIT license: + + 0004-disable-geoip-enrich.patch + * Ship a selection of hub files in /usr/share/crowdsec/hub so that + crowdsec can be set up without having to download data from the + collaborative hub (https://hub.crowdsec.net/). + * Ditto for some data files (in /usr/share/crowdsec/data). + * Use DH_GOLANG_EXCLUDES to avoid including extra Go files from the + hub into the build directory. + * Implement an extract_hub_tarball target in debian/rules to automate + extracting hub files from the tarball. + * Implement an extract_data_tarball target in debian/rules to automate + extracting data files from the tarball. + * Ship crowdsec-cli (automated Golang naming) as cscli (upstream's + preference). + * Add patch to adjust the default config: + + 0005-adjust-config.patch + * Ship config/config.yaml accordingly, along with the config files it + references. + * Also adjust the hub_branch variable in config.yaml, pointing to the + branch related to the current upstream release instead of master. + * Create /var/lib/crowdsec/{data,hub} directories. + * Implement configure in postinst to generate credentials files: + Implement a simple agent setup with a Local API (LAPI), and with an + automatic registration to the Central API (CAPI). The latter can be + disabled by creating a /etc/crowdsec/online_api_credentials.yaml file + containing a comment (e.g. “# no thanks”) before installing this + package. + * Implement purge in postrm. Drop all of /etc/crowdsec except + online_api_credentials.yaml if this file doesn't seem to have been + created during CAPI registration (likely because an admin created the + file in advance to prevent it). Also remove everything below + /var/lib/crowdsec/{data,hub}, along with log files. + * Implement custom enable-online-hub and disable-online-hub actions in + postinst. The latter is called once automatically to make sure the + offline hub is ready to use. See README.Debian for details. + * Also enable all items using the offline hub on fresh installation. + * Add patch advertising `systemctl restart crowdsec` when updating the + configuration: reload doesn't work at the moment (#656 upstream). + + 0006-prefer-systemctl-restart.patch + * Add patch automating switching from the offline hub to the online hub + when `cscli hub update` is called: + + 0007-automatically-enable-online-hub.patch + * Add lintian override accordingly: uses-dpkg-database-directly. + * Add ca-certificates to Depends for the CAPI registration. + * Create /etc/machine-id if it doesn't exist already (e.g. in piuparts + environments). + + -- Cyril Brulebois Tue, 02 Mar 2021 00:25:48 +0000 + +crowdsec (1.0.4-1) unstable; urgency=medium + + * New upstream release. + * Bump copyright years. + * Bump golang-github-facebook-ent-dev build-dep. + * Swap Maintainer/Uploaders: the current plan is for me to keep in touch + with upstream to coordinate packaging work in Debian. Help from fellow + members of the Debian Go Packaging Team is very welcome, though! + * Fix typos in the long description, and merge upstream's review. + * Refresh patch: + + 0001-use-a-local-machineid-implementation.patch + * Drop patch (merged upstream): + + 1001-fix-docker-container-creation-for-metabase-563.patch + + -- Cyril Brulebois Wed, 03 Feb 2021 08:54:24 +0000 + +crowdsec (1.0.2-1) unstable; urgency=medium + + * Initial release (Closes: #972573): start by shipping binaries, + while better integration is being worked on with upstream: + documentation and assisted configuration are coming up. + * Version some build-deps as earlier versions are known not to work. + * Use a local machineid implementation instead of depending on an + extra package: + + 0001-use-a-local-machineid-implementation.patch + * Use a syntax that's compatible with version 1.6.0 of the sqlite3 + driver: + + 0002-add-compatibility-for-older-sqlite-driver.patch + * Backport upstream fix for golang-github-docker-docker-dev version + currently in unstable: + + 1001-fix-docker-container-creation-for-metabase-563.patch + * Install all files in the build directory so that the testsuite finds + required test data that's scattered all over the place. + * Add systemd to Build-Depends for the testsuite, so that it finds + the journalctl binary. + * Add lintian overrides for the hardening-no-pie warnings: PIE is not + relevant for Go packages. + + -- Cyril Brulebois Thu, 14 Jan 2021 02:46:18 +0000 diff --git a/control b/control new file mode 100644 index 0000000..9e88faa --- /dev/null +++ b/control @@ -0,0 +1,193 @@ +Source: crowdsec +Maintainer: Cyril Brulebois +Uploaders: Debian Go Packaging Team +Section: golang +Testsuite: autopkgtest-pkg-go +Priority: optional +Build-Depends: debhelper-compat (= 13), + dh-golang, + golang-any, + golang-entgo-ent-dev, + golang-github-alecaivazis-survey-dev, + golang-github-alexliesenfeld-health-dev, + golang-github-antonmedv-expr-dev, + golang-github-appleboy-gin-jwt-dev, + golang-github-aquasecurity-table-dev, + golang-github-beevik-etree-dev, + golang-github-blackfireio-osinfo-dev, + golang-github-buger-jsonparser-dev, + golang-github-c-robinson-iplib-dev, + golang-github-confluentinc-bincover-dev, + golang-github-coreos-go-systemd-dev, + golang-github-crowdsecurity-dlog-dev, + golang-github-crowdsecurity-grokky-dev, + golang-github-crowdsecurity-machineid-dev, + golang-github-davecgh-go-spew-dev, + golang-github-dghubble-sling-dev, + golang-github-docker-docker-dev, + golang-github-docker-go-connections-dev, + golang-github-enescakir-emoji-dev, + golang-github-gin-gonic-gin-dev (>= 1.8.1), + golang-github-go-co-op-gocron-dev, + golang-github-go-openapi-errors-dev, + golang-github-go-openapi-strfmt-dev, + golang-github-go-openapi-swag-dev, + golang-github-go-openapi-validate-dev, + golang-github-go-sql-driver-mysql-dev, + golang-github-google-go-querystring-dev, + golang-github-goombaio-namegenerator-dev, + golang-github-hashicorp-go-plugin-dev, + golang-github-hashicorp-go-version-dev, + golang-github-ivanpirog-coloredcobra-dev, + golang-github-jackc-pgx-v4-dev, + golang-github-jarcoal-httpmock-dev, + golang-github-jszwec-csvutil-dev, + golang-github-lib-pq-dev, + golang-github-masterminds-sprig-dev, + golang-github-mattn-go-sqlite3-dev, + golang-github-mohae-deepcopy-dev, + golang-github-nxadm-tail-dev, + golang-github-opencontainers-image-spec-dev, + golang-github-oschwald-geoip2-golang-dev (>= 1.2), + golang-github-oschwald-maxminddb-golang-dev (>= 1.4), + golang-github-pbnjay-memory-dev, + golang-github-pkg-errors-dev, + golang-github-prometheus-client-golang-dev, + golang-github-prometheus-client-model-dev, + golang-github-prometheus-prom2json-dev, + golang-github-r3labs-diff-dev, + golang-github-sirupsen-logrus-dev, + golang-github-slack-go-slack-dev, + golang-github-spf13-cobra-dev, + golang-github-stretchr-testify-dev, + golang-github-texttheater-golang-levenshtein-dev, + golang-github-xhit-go-simple-mail-dev, + golang-golang-x-crypto-dev, + golang-golang-x-mod-dev, + golang-golang-x-sys-dev, + golang-gopkg-natefinch-lumberjack.v2-dev, + golang-gopkg-tomb.v2-dev, + golang-gopkg-yaml.v2-dev, + python3, + systemd +Standards-Version: 4.5.0 +Vcs-Browser: https://salsa.debian.org/go-team/packages/crowdsec +Vcs-Git: https://salsa.debian.org/go-team/packages/crowdsec.git +Homepage: https://github.com/crowdsecurity/crowdsec +Rules-Requires-Root: no +XS-Go-Import-Path: github.com/crowdsecurity/crowdsec + +Package: crowdsec +Architecture: any +Depends: ca-certificates, + ${misc:Depends}, + ${shlibs:Depends}, + libsqlite3-0 (>= 3.35.0), +Built-Using: ${misc:Built-Using} +Description: lightweight and collaborative security engine + CrowdSec is a lightweight security engine, able to detect and remedy + aggressive network behavior. It can leverage and also enrich a + global community-wide IP reputation database, to help fight online + cybersec aggressions in a collaborative manner. + . + CrowdSec can read many log sources, parse and also enrich them, in + order to detect specific scenarios, that usually represent malevolent + behavior. Parsers, Enrichers, and Scenarios are YAML files that can + be shared and downloaded through a specific Hub, as well as be created + or adapted locally. + . + Detection results are available for CrowdSec, its CLI tools and + bouncers via an HTTP API. Triggered scenarios lead to an alert, which + often results in a decision (e.g. IP banned for 4 hours) that can be + consumed by bouncers (software components enforcing a decision, such + as an iptables ban, an nginx lua script, or any custom user script). + . + The CLI allows users to deploy a Metabase Docker image to provide + simple-to-deploy dashboards of ongoing activity. The CrowdSec daemon + is also instrumented with Prometheus to provide observability. + . + CrowdSec can be used against live logs (“à la fail2ban”), but can + also work on cold logs to help, in a forensic context, to build an + analysis for past events. + . + On top of that, CrowdSec aims at sharing detection signals amongst + all participants, to pre-emptively allow users to block likely + attackers. To achieve this, minimal meta-information about the attack + is shared with the CrowdSec organization for further retribution. + . + Users can also decide not to take part into the collective effort via + the central API, but to register on a local API instead. + +Package: golang-github-crowdsecurity-crowdsec-dev +Architecture: all +Multi-Arch: foreign +Depends: golang-entgo-ent-dev, + golang-github-alecaivazis-survey-dev, + golang-github-alexliesenfeld-health-dev, + golang-github-antonmedv-expr-dev, + golang-github-appleboy-gin-jwt-dev, + golang-github-aquasecurity-table-dev, + golang-github-beevik-etree-dev, + golang-github-blackfireio-osinfo-dev, + golang-github-buger-jsonparser-dev, + golang-github-c-robinson-iplib-dev, + golang-github-confluentinc-bincover-dev, + golang-github-coreos-go-systemd-dev, + golang-github-crowdsecurity-dlog-dev, + golang-github-crowdsecurity-grokky-dev, + golang-github-crowdsecurity-machineid-dev, + golang-github-davecgh-go-spew-dev, + golang-github-dghubble-sling-dev, + golang-github-docker-docker-dev, + golang-github-docker-go-connections-dev, + golang-github-enescakir-emoji-dev, + golang-github-gin-gonic-gin-dev (>= 1.8.1), + golang-github-go-co-op-gocron-dev, + golang-github-go-openapi-errors-dev, + golang-github-go-openapi-strfmt-dev, + golang-github-go-openapi-swag-dev, + golang-github-go-openapi-validate-dev, + golang-github-go-sql-driver-mysql-dev, + golang-github-google-go-querystring-dev, + golang-github-goombaio-namegenerator-dev, + golang-github-hashicorp-go-plugin-dev, + golang-github-hashicorp-go-version-dev, + golang-github-ivanpirog-coloredcobra-dev, + golang-github-jackc-pgx-v4-dev, + golang-github-jarcoal-httpmock-dev, + golang-github-jszwec-csvutil-dev, + golang-github-lib-pq-dev, + golang-github-masterminds-sprig-dev, + golang-github-mattn-go-sqlite3-dev, + golang-github-mohae-deepcopy-dev, + golang-github-nxadm-tail-dev, + golang-github-opencontainers-image-spec-dev, + golang-github-oschwald-geoip2-golang-dev (>= 1.2), + golang-github-oschwald-maxminddb-golang-dev (>= 1.4), + golang-github-pbnjay-memory-dev, + golang-github-pkg-errors-dev, + golang-github-prometheus-client-golang-dev, + golang-github-prometheus-client-model-dev, + golang-github-prometheus-prom2json-dev, + golang-github-r3labs-diff-dev, + golang-github-sirupsen-logrus-dev, + golang-github-slack-go-slack-dev, + golang-github-spf13-cobra-dev, + golang-github-stretchr-testify-dev, + golang-github-texttheater-golang-levenshtein-dev, + golang-github-xhit-go-simple-mail-dev, + golang-golang-x-crypto-dev, + golang-golang-x-mod-dev, + golang-golang-x-sys-dev, + golang-gopkg-natefinch-lumberjack.v2-dev, + golang-gopkg-tomb.v2-dev, + golang-gopkg-yaml.v2-dev, + python3, + ${misc:Depends} +Description: lightweight and collaborative security engine - library + CrowdSec is a lightweight security engine, able to detect and remedy + aggressive network behavior. It can leverage and also enrich a + global community-wide IP reputation database, to help fight online + cybersec aggressions in a collaborative manner. + . + This package contains the development files. diff --git a/copyright b/copyright new file mode 100644 index 0000000..37fb698 --- /dev/null +++ b/copyright @@ -0,0 +1,74 @@ +Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/ +Upstream-Name: crowdsec +Upstream-Contact: contact@crowdsec.net +Source: https://github.com/crowdsecurity/crowdsec + +Files: * +Copyright: 2020-2021 crowdsecurity +License: Expat + +Files: pkg/time/* +Copyright: 2009-2015 The Go Authors + 2020 crowdsecurity +License: BSD-3 +Comment: improved version of x/time/rate + +Files: data*/bad_user_agents.txt +Copyright: 2017 Mitchell Krog +License: Expat + +Files: hub*/parsers/s01-parse/crowdsecurity/postfix-logs.yaml +Copyright: 2014, 2015 Rudy Gevaert + 2020 Crowdsec +License: Expat + +Files: debian/* +Copyright: 2020-2021 Cyril Brulebois +License: Expat +Comment: Debian packaging is licensed under the same terms as upstream + +License: Expat + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + . + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + . + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE. + +License: BSD-3 + Redistribution and use in source and binary forms, with or without + modification, are permitted provided that the following conditions are + met: + . + * Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. + * Redistributions in binary form must reproduce the above + copyright notice, this list of conditions and the following disclaimer + in the documentation and/or other materials provided with the + distribution. + * Neither the name of Google Inc. nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + . + THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. diff --git a/crowdsec.conffiles b/crowdsec.conffiles new file mode 100644 index 0000000..98d3eb6 --- /dev/null +++ b/crowdsec.conffiles @@ -0,0 +1 @@ +remove-on-upgrade /etc/logrotate.d/crowdsec diff --git a/crowdsec.dirs b/crowdsec.dirs new file mode 100644 index 0000000..3970fa5 --- /dev/null +++ b/crowdsec.dirs @@ -0,0 +1,2 @@ +/var/lib/crowdsec/data +/var/lib/crowdsec/hub diff --git a/crowdsec.install b/crowdsec.install new file mode 100644 index 0000000..3857019 --- /dev/null +++ b/crowdsec.install @@ -0,0 +1,25 @@ +# Main config: +config/config.yaml etc/crowdsec/ +# Referenced configs: +config/acquis.yaml etc/crowdsec/ +config/profiles.yaml etc/crowdsec/ +config/simulation.yaml etc/crowdsec/ + +config/patterns/* etc/crowdsec/patterns +config/crowdsec.service lib/systemd/system +hub*/blockers usr/share/crowdsec/hub +hub*/collections usr/share/crowdsec/hub +hub*/parsers usr/share/crowdsec/hub +hub*/postoverflows usr/share/crowdsec/hub +hub*/scenarios usr/share/crowdsec/hub +hub*/.index.json usr/share/crowdsec/hub +data*/* usr/share/crowdsec/data + +usr/bin/crowdsec +usr/bin/crowdsec-cli + +usr/bin/dummy usr/lib/crowdsec/plugins +usr/bin/email usr/lib/crowdsec/plugins +usr/bin/http usr/lib/crowdsec/plugins +usr/bin/slack usr/lib/crowdsec/plugins +usr/bin/splunk usr/lib/crowdsec/plugins diff --git a/crowdsec.lintian-overrides b/crowdsec.lintian-overrides new file mode 100644 index 0000000..8b2ea92 --- /dev/null +++ b/crowdsec.lintian-overrides @@ -0,0 +1,10 @@ +# PIE is not relevant for Go packages (for reference, lintian's +# $built_with_golang variable is the one that's not set properly +# for this package, meaning this tag is emitted regardless): +crowdsec: hardening-no-pie [usr/bin/crowdsec] +crowdsec: hardening-no-pie [usr/bin/cscli] +crowdsec: hardening-no-pie [usr/lib/crowdsec/plugins/*] + +# The postinst script implements custom actions, sharing code with the +# "configure" one: +crowdsec: uses-dpkg-database-directly [usr/bin/cscli] diff --git a/crowdsec.postinst b/crowdsec.postinst new file mode 100644 index 0000000..63af669 --- /dev/null +++ b/crowdsec.postinst @@ -0,0 +1,314 @@ +#!/bin/sh +set -e + +# See README.Debian for the distinction between online and offline +# hubs: +OFFLINE_HUB=/usr/share/crowdsec/hub +LIVE_HUB=/var/lib/crowdsec/hub +ITEMS="blockers collections parsers postoverflows scenarios .index.json" + +# Offline hub = symlinks are in place, so that an updated Debian +# package ships updated items from the hub: +disable_online_hub() { + rm -rf "$LIVE_HUB" + mkdir "$LIVE_HUB" + for item in $ITEMS; do + ln -s "$OFFLINE_HUB/$item" "$LIVE_HUB" + done +} + +# Online hub = we replace symlinks with a copy of the items they point +# to, so that enabled items (symlinks from /etc) aren't disabled +# because of dangling symlinks. Let `cscli hub upgrade` replace the +# original copy as required: +enable_online_hub() { + # Idempotence: once this function has been called once, .index.json + # should no longer be a symlink, so it can be called each time + # `cscli hub update` is called: + if [ -L "$LIVE_HUB/.index.json" ]; then + echo "I: Switching from offline hub to online hub (see README.Debian)" >&2 + for item in $ITEMS; do + if [ -L "$LIVE_HUB/$item" ]; then + rm -f "$LIVE_HUB/$item" + cp -r "$OFFLINE_HUB/$item" "$LIVE_HUB" + fi + done + fi +} + +# Only enable a few collections, as agreed with upstream: +UPSTREAM_COLLECTIONS="crowdsecurity/linux crowdsecurity/apache2 crowdsecurity/nginx" + +# The following items are everything they need (including other collections), plus +# the whitelists parser: +UPSTREAM_ITEMS=$(cat <&2 + touch "$LAPI" + # This is required as of 1.0.8 at least: + touch "$CAPI" + + # Minimal environments (e.g. piuparts): + if [ ! -f /etc/machine-id ]; then + echo "W: Missing /etc/machine-id, initializing" >&2 + sed 's/-//g' < /proc/sys/kernel/random/uuid > /etc/machine-id + fi + + cscli --error machines add --force "$(cat /etc/machine-id)" --password "$(tr -dc 'a-zA-Z0-9' < /dev/urandom | fold -w 32 | head -n 1)" + fi + + # Heuristics: if the file is empty, it's probably been just created + # by the touch call above, and we want to register. Otherwise, + # either the user has created a file in advance to disable CAPI + # registration, or we've already registered to CAPI in a previous + # configure run (in both cases, don't do anything): + if [ ! -s "$CAPI" ]; then + echo "I: Registering to CAPI ($CAPI)" >&2 + cscli --error capi register + fi + + # Missing index means initial install, let's go for setting up + # offline hub + enabling everything per upstream recommendation: + if [ ! -e /var/lib/crowdsec/hub/.index.json ]; then + echo "I: Setting up offline hub (see README.Debian)" >&2 + disable_online_hub + + # Symlinks: + echo "I: Enabling upstream-recommended items, first installation (via symlinks from /etc/crowdsec)" >&2 + find /var/lib/crowdsec/hub/*/ -name '*yaml' | \ + while read target; do + source=${target##/var/lib/crowdsec/hub/} + # Match upstream's recommendations: + if ! echo "$UPSTREAM_ITEMS" | grep -qs "^$source$"; then + continue + fi + + # Code is picky about the number of (sub)directories, so the + # vendor must be stripped (seen in 1.0.*, 1.4.*): + source=$(dirname $(dirname "$source"))/$(basename "$source") + mkdir -p /etc/crowdsec/$(dirname "$source") + ln -s "$target" "/etc/crowdsec/$source" + done + + # Initial copy of data files: + cp /usr/share/crowdsec/data/* /var/lib/crowdsec/data/ + elif [ -L /var/lib/crowdsec/hub/.index.json ]; then + # Check whether upstream recommended collections are still all present. If + # they are, make sure their dependencies are deployed. If one of them is + # missing, the admin touched something, so don't touch anything: + found=0 + n_collection=$(echo "$UPSTREAM_COLLECTIONS" | wc -w) + for collection in $UPSTREAM_COLLECTIONS; do + link=/etc/crowdsec/collections/$(basename $collection).yaml + if [ -f "$link" ] && readlink "$link" | grep -qs "$collection\.yaml$"; then + found=$((found+1)) + fi + done + # When upgrading from 1.4.6-1(+bN) specifically, assume we need to fix + # enabling the upstream-recommended collections: + if [ "$found" = "$n_collection" ] || \ + echo "$2" | grep -qs '^1\.4\.6-1\(\|+b[0-9][0-9]*\)$'; then + echo "I: All $n_collection upstream-recommended collections are active, updating their dependencies" >&2 + + find /var/lib/crowdsec/hub/*/ -name '*yaml' | \ + while read target; do + source=${target##/var/lib/crowdsec/hub/} + # Match upstream's recommendations: + if ! echo "$UPSTREAM_ITEMS" | grep -qs "^$source$"; then + continue + fi + + # Code is picky about the number of (sub)directories, so the + # vendor must be stripped (seen in 1.0.*, 1.4.*): + source=$(dirname $(dirname "$source"))/$(basename "$source") + mkdir -p /etc/crowdsec/$(dirname "$source") + ln -sf "$target" "/etc/crowdsec/$source" + done + else + echo "I: $found/$n_collection upstream-recommended collections are active, not touching collections" >&2 + fi + + # Refresh data files: + cp /usr/share/crowdsec/data/* /var/lib/crowdsec/data/ + else + # Upgrade with online hub, leave everything up to the admin + : + fi +fi + +# Best effort guess to avoid performance issues with SQLite and warnings: +CONFIG=/etc/crowdsec/config.yaml + +if [ "$1" = configure ] && [ ! -f "$CONFIG.local" ]; then + db_type=$(cscli config show --key Config.DbConfig.Type) + db_path=$(cscli config show --key Config.DbConfig.DbPath) + db_wal=$(cscli config show --key Config.DbConfig.UseWal) + # Unconfigured is likely but let's focus on neither true nor false, + # which might be more stable across versions: + if [ "$db_type" = sqlite ] && [ "$db_wal" != true ] && [ "$db_wal" != false ]; then + path=$db_path + while [ -z "$fstype" ] && [ "$path" != / ]; do + path=$(dirname "$path") + fstype=$(findmnt --noheadings --output FSTYPE "$path" || true) + done + case "$fstype" in + nfs*) + echo "I: Disabling WAL for SQLite [fstype=$fstype] (see README.Debian)" >&2 + use_wal=false + ;; + *) + echo "I: Enabling WAL for SQLite [fstype=$fstype] (see README.Debian)" >&2 + use_wal=true + ;; + esac + cat > "$CONFIG.local" <&2 + ;; + enable-online-hub) + enable_online_hub + ;; +esac + +# This is a workaround when upgrading from old versions (1.0.9* in bullseye), +# which take a long time to shut down (#1031326), lowering the timeout from +# the default 1m 30s to just 20s (the dh_installsystemd-generated code takes +# cares of daemon-reload already): +if [ "$1" = configure ] && [ -n "$2" ] && dpkg --compare-versions "$2" '<<' 1.4 && [ -d /run/systemd/system ]; then + echo "W: Lowering TimeoutStopSec to 20 seconds (#1031326)" >&2 + mkdir -p /run/systemd/system/crowdsec.service.d + cat >/run/systemd/system/crowdsec.service.d/override.conf <&2 + deb-systemd-invoke restart 'crowdsec.service' >/dev/null || true + fi +fi + +# This is a workaround for race conditions during the first start-up, filed +# upstream as . The +# systemd logic (/run/systemd/system and deb-systemd-invoke) is adapted from +# dh_installsystemd-generated code: +if [ "$1" = configure ] && [ -z "$2" ] && [ -d /run/systemd/system ]; then + # Exit early if the unit isn't active: + if ! systemctl is-active crowdsec.service >/dev/null 2>&1; then + exit 0 + fi + + LOG=/var/log/crowdsec.log + MAX=20 + for _ in $(seq 1 $MAX); do + # Getting decisions means we can happily exit: + if grep -qs 'added [0-9][0-9]* entries, deleted [0-9][0-9]* entries' $LOG; then + break + fi + # Getting 0 new entries means we can happily trigger a restart then exit: + if grep -qs 'received 0 new entries (expected if you just installed crowdsec)' $LOG; then + echo "W: Restarting manually to force a CAPI pull (upstream #2120)" >&2 + deb-systemd-invoke restart 'crowdsec.service' >/dev/null || true + break + fi + # Don't poll too aggressively: + sleep 1 + done +fi + +# Bouncer registration: they have crowdsec in Recommends only, so ordering isn't +# guaranteed (#1035499, #1036985). Process pending registration if any, then +# kick relevant systemd units once their ConditionPathExists is satisfied. +PENDING=/var/lib/crowdsec/pending-registration +if [ -f $PENDING ]; then + while read unit name key; do + units="${units:+$units }$unit" + bouncers="${bouncers:+$bouncers }$name" + # We don't need the API key to be echo'd back: + cscli --error -oraw bouncers add "$name" -k "$key" > /dev/null + done < $PENDING + rm -f $PENDING + echo "I: Registered bouncers: $bouncers" >&2 + + for unit in $units; do + deb-systemd-invoke start "$unit" + done + echo "I: Restarts units: $units" >&2 +fi diff --git a/crowdsec.postrm b/crowdsec.postrm new file mode 100644 index 0000000..634f495 --- /dev/null +++ b/crowdsec.postrm @@ -0,0 +1,48 @@ +#!/bin/sh +set -e + +CAPI=/etc/crowdsec/online_api_credentials.yaml +LAPI=/etc/crowdsec/local_api_credentials.yaml + +if [ "$1" = purge ]; then + # The CAPI config file might have been created by the postinst during CAPI + # registration, or created by the admin to prevent CAPI registration. Make + # sure to keep it in the latter case. + # + # Also, don't touch the bouncers directory, it's perfectly fine to install + # bouncers without crowdsec. + find /etc/crowdsec -mindepth 1 -maxdepth 1 | sort | while read path; do + if [ "$path" = "$CAPI" ]; then + if ! grep -qs '^url: https://api.crowdsec.net/$' "$CAPI"; then + echo "W: not purging $path" >&2 + continue + fi + elif [ "$path" = /etc/crowdsec/bouncers ]; then + echo "W: not purging $path" >&2 + continue + fi + rm -rf "$path" + done + rmdir --ignore-fail-on-non-empty /etc/crowdsec + + # Local config and hub: + rm -rf /var/lib/crowdsec/data + rm -rf /var/lib/crowdsec/hub + + # Logs: + # - main logs and possible logrotate-generated logs (obsolete starting + # with 1.4.6-2 but files might have been created before): + rm -f /var/log/crowdsec.log* + rm -f /var/log/crowdsec_api.log* + # - internal rotation (lumberjack.Logger), not matching the following + # format exactly to avoid an extra long pattern, but matching what + # looks like a date and a time to avoid removing bouncer logs (as + # crowdsec-{firewall,custom}-bouncer.log would match crowdsec-*.log): + # + # backupTimeFormat = "2006-01-02T15-04-05.000" + for prefix in crowdsec crowdsec_api; do + find /var/log/ -name "${prefix}-[0-9-]*T[0-9.-]*.log*" -delete + done +fi + +#DEBHELPER# diff --git a/gbp.conf b/gbp.conf new file mode 100644 index 0000000..3d450c2 --- /dev/null +++ b/gbp.conf @@ -0,0 +1,3 @@ +[DEFAULT] +debian-branch = debian/sid +dist = DEP14 diff --git a/gitlab-ci.yml b/gitlab-ci.yml new file mode 100644 index 0000000..594e14e --- /dev/null +++ b/gitlab-ci.yml @@ -0,0 +1,6 @@ +# auto-generated, DO NOT MODIFY. +# The authoritative copy of this file lives at: +# https://salsa.debian.org/go-team/infra/pkg-go-tools/blob/master/config/gitlabciyml.go +--- +include: + - https://salsa.debian.org/go-team/infra/pkg-go-tools/-/raw/master/pipeline/test-archive.yml diff --git a/golang-github-crowdsecurity-crowdsec-dev.install b/golang-github-crowdsecurity-crowdsec-dev.install new file mode 100644 index 0000000..dd222db --- /dev/null +++ b/golang-github-crowdsecurity-crowdsec-dev.install @@ -0,0 +1 @@ +usr/share diff --git a/golang-github-crowdsecurity-crowdsec-dev.lintian-overrides b/golang-github-crowdsecurity-crowdsec-dev.lintian-overrides new file mode 100644 index 0000000..6b8919f --- /dev/null +++ b/golang-github-crowdsecurity-crowdsec-dev.lintian-overrides @@ -0,0 +1,9 @@ +# False positive (bash and extglob): +golang-github-crowdsecurity-crowdsec-dev: shell-script-fails-syntax-check [usr/share/gocode/src/github.com/crowdsecurity/crowdsec/docker/docker_start.sh] + +# False positive (test file, purposefully broken): +golang-github-crowdsecurity-crowdsec-dev: broken-gz [usr/share/gocode/src/github.com/crowdsecurity/crowdsec/pkg/acquisition/modules/file/test_files/bad.gz] + +# Test files: +golang-github-crowdsecurity-crowdsec-dev: script-not-executable [usr/share/gocode/src/github.com/crowdsecurity/crowdsec/tests/bats/*.bats] +golang-github-crowdsecurity-crowdsec-dev: unusual-interpreter bats [usr/share/gocode/src/github.com/crowdsecurity/crowdsec/tests/bats/*.bats] diff --git a/patches/0003-adjust-systemd-unit.patch b/patches/0003-adjust-systemd-unit.patch new file mode 100644 index 0000000..d04b865 --- /dev/null +++ b/patches/0003-adjust-systemd-unit.patch @@ -0,0 +1,32 @@ +From: Cyril Brulebois +Date: Fri, 22 Jan 2021 13:25:54 +0000 +Subject: Adjust systemd unit + + - Adjust paths for the packaged crowdsec binary (/usr/bin). + - Drop commented out ExecStartPost entirely. + - Drop syslog.target dependency, it's socket-activated (thanks to the + systemd-service-file-refers-to-obsolete-target lintian tag). + - Ensure both local and online API credentials have been defined. + +--- a/config/crowdsec.service ++++ b/config/crowdsec.service +@@ -1,13 +1,15 @@ + [Unit] + Description=Crowdsec agent +-After=syslog.target network.target remote-fs.target nss-lookup.target ++After=network.target remote-fs.target nss-lookup.target ++# Ensure configuration happened: ++ConditionPathExists=/etc/crowdsec/local_api_credentials.yaml ++ConditionPathExists=/etc/crowdsec/online_api_credentials.yaml + + [Service] + Type=notify + Environment=LC_ALL=C LANG=C +-ExecStartPre=/usr/local/bin/crowdsec -c /etc/crowdsec/config.yaml -t +-ExecStart=/usr/local/bin/crowdsec -c /etc/crowdsec/config.yaml +-#ExecStartPost=/bin/sleep 0.1 ++ExecStartPre=/usr/bin/crowdsec -c /etc/crowdsec/config.yaml -t ++ExecStart=/usr/bin/crowdsec -c /etc/crowdsec/config.yaml + ExecReload=/bin/kill -HUP $MAINPID + + [Install] diff --git a/patches/0004-disable-geoip-enrich.patch b/patches/0004-disable-geoip-enrich.patch new file mode 100644 index 0000000..dd88067 --- /dev/null +++ b/patches/0004-disable-geoip-enrich.patch @@ -0,0 +1,147 @@ +From: Cyril Brulebois +Date: Fri, 22 Jan 2021 14:35:42 +0000 +Subject: Disable geoip-enrich in the hub files + +It would download GeoLite2*.mmdb files from the network. Let users +enable the hub by themselves if they want to use it. + +When refreshing this patch, don't forget to update both digest and +content fields, using: + + - digest: sha256sum hub1/collections/crowdsecurity/linux.yaml + - content: base64 -w 0 /etc/crowdsec/collections/linux.yaml + +--- a/hub1/.index.json ++++ b/hub1/.index.json +@@ -585,12 +585,11 @@ + }, + "long_description": "Kipjb3JlIHBhY2thZ2UgZm9yIGZyZWVic2QqKgoKY29udGFpbnMgc3VwcG9ydCBmb3Igc3lzbG9nLCBkbyBub3QgcmVtb3ZlLgo=", + "content": "cGFyc2VyczoKICAtIGNyb3dkc2VjdXJpdHkvc3lzbG9nLWxvZ3MKICAtIGNyb3dkc2VjdXJpdHkvZ2VvaXAtZW5yaWNoCiAgLSBjcm93ZHNlY3VyaXR5L2RhdGVwYXJzZS1lbnJpY2gKY29sbGVjdGlvbnM6CiAgLSBjcm93ZHNlY3VyaXR5L3NzaGQKZGVzY3JpcHRpb246ICJjb3JlIGZyZWVic2Qgc3VwcG9ydCA6IHN5c2xvZytnZW9pcCtzc2giCmF1dGhvcjogY3Jvd2RzZWN1cml0eQp0YWdzOgogIC0gZnJlZWJzZCAKCg==", +- "description": "core freebsd support : syslog+geoip+ssh", ++ "description": "core freebsd support : syslog+ssh", + "author": "crowdsecurity", + "labels": null, + "parsers": [ + "crowdsecurity/syslog-logs", +- "crowdsecurity/geoip-enrich", + "crowdsecurity/dateparse-enrich" + ], + "collections": [ +@@ -819,18 +818,17 @@ + "deprecated": false + }, + "0.2": { +- "digest": "baaa37b12b4d734fab81ae01ff81c58ceb7a99304f21e6bb6ff86b871ed6d5eb", ++ "digest": "21ac34a4e2146ac8cd42f8377e1af5ead7eef5447bf3d6b0bf4e8ca456a7c16d", + "deprecated": false + } + }, + "long_description": "Kipjb3JlIHBhY2thZ2UgZm9yIGxpbnV4KioKCmNvbnRhaW5zIHN1cHBvcnQgZm9yIHN5c2xvZywgZG8gbm90IHJlbW92ZS4K", +- "content": "cGFyc2VyczoKICAtIGNyb3dkc2VjdXJpdHkvc3lzbG9nLWxvZ3MKICAtIGNyb3dkc2VjdXJpdHkvZ2VvaXAtZW5yaWNoCiAgLSBjcm93ZHNlY3VyaXR5L2RhdGVwYXJzZS1lbnJpY2gKY29sbGVjdGlvbnM6CiAgLSBjcm93ZHNlY3VyaXR5L3NzaGQKZGVzY3JpcHRpb246ICJjb3JlIGxpbnV4IHN1cHBvcnQgOiBzeXNsb2crZ2VvaXArc3NoIgphdXRob3I6IGNyb3dkc2VjdXJpdHkKdGFnczoKICAtIGxpbnV4Cgo=", +- "description": "core linux support : syslog+geoip+ssh", ++ "content": "cGFyc2VyczoKICAtIGNyb3dkc2VjdXJpdHkvc3lzbG9nLWxvZ3MKICAtIGNyb3dkc2VjdXJpdHkvZGF0ZXBhcnNlLWVucmljaApjb2xsZWN0aW9uczoKICAtIGNyb3dkc2VjdXJpdHkvc3NoZApkZXNjcmlwdGlvbjogImNvcmUgbGludXggc3VwcG9ydCA6IHN5c2xvZytzc2giCmF1dGhvcjogY3Jvd2RzZWN1cml0eQp0YWdzOgogIC0gbGludXgKCg==", ++ "description": "core linux support : syslog+ssh", + "author": "crowdsecurity", + "labels": null, + "parsers": [ + "crowdsecurity/syslog-logs", +- "crowdsecurity/geoip-enrich", + "crowdsecurity/dateparse-enrich" + ], + "collections": [ +@@ -902,8 +900,7 @@ + "parsers": [ + "crowdsecurity/syslog-logs", + "crowdsecurity/magento-extension-logs", +- "crowdsecurity/dateparse-enrich", +- "crowdsecurity/geoip-enrich" ++ "crowdsecurity/dateparse-enrich" + ], + "scenarios": [ + "crowdsecurity/http-magento-bf", +@@ -1473,7 +1470,6 @@ + "parsers": [ + "crowdsecurity/windows-logs", + "crowdsecurity/windows-auth", +- "crowdsecurity/geoip-enrich", + "crowdsecurity/dateparse-enrich" + ], + "scenarios": [ +@@ -2532,26 +2528,6 @@ + "author": "crowdsecurity", + "labels": null + }, +- "crowdsecurity/geoip-enrich": { +- "path": "parsers/s02-enrich/crowdsecurity/geoip-enrich.yaml", +- "stage": "s02-enrich", +- "version": "0.2", +- "versions": { +- "0.1": { +- "digest": "c0718adfc71ad462ad90485ad5c490e5de0e54d8af425bff552994e114443ab6", +- "deprecated": false +- }, +- "0.2": { +- "digest": "ab327e6044a32de7d2f3780cbc8e0c4af0c11716f353023d2dc7b986571bb765", +- "deprecated": false +- } +- }, +- "long_description": "VGhlIEdlb0lQIG1vZHVsZSByZWxpZXMgb24gZ2VvbGl0ZSBkYXRhYmFzZSB0byBwcm92aWRlIGVucmljaG1lbnQgb24gc291cmNlIGlwLgoKVGhlIGZvbGxvd2luZyBpbmZvcm1hdGlvbnMgd2lsbCBiZSBhZGRlZCB0byB0aGUgZXZlbnQgOgogLSBgTWV0YS5Jc29Db2RlYCA6IHR3by1sZXR0ZXJzIGNvdW50cnkgY29kZQogLSBgTWV0YS5Jc0luRVVgIDogYSBib29sZWFuIGluZGljYXRpbmcgaWYgSVAgaXMgaW4gRVUKIC0gYE1ldGEuR2VvQ29vcmRzYCA6IGxhdGl0dWRlICYgbG9uZ2l0dWRlIG9mIElQCiAtIGBNZXRhLkFTTk51bWJlcmAgOiBBdXRvbm9tb3VzIFN5c3RlbSBOdW1iZXIKIC0gYE1ldGEuQVNOT3JnYCA6IEF1dG9ub21vdXMgU3lzdGVtIE5hbWUKIC0gYE1ldGEuU291cmNlUmFuZ2VgIDogVGhlIHB1YmxpYyByYW5nZSB0byB3aGljaCB0aGUgSVAgYmVsb25ncwoKClRoaXMgY29uZmlndXJhdGlvbiBpbmNsdWRlcyBHZW9MaXRlMiBkYXRhIGNyZWF0ZWQgYnkgTWF4TWluZCBhdmFpbGFibGUgZnJvbSBbaHR0cHM6Ly93d3cubWF4bWluZC5jb21dKGh0dHBzOi8vd3d3Lm1heG1pbmQuY29tKSwgaXQgaW5jbHVkZXMgdHdvIGRhdGEgZmlsZXM6IAoqIFtHZW9MaXRlMi1DaXR5Lm1tZGJdKGh0dHBzOi8vY3Jvd2RzZWMtc3RhdGljcy1hc3NldHMuczMtZXUtd2VzdC0xLmFtYXpvbmF3cy5jb20vR2VvTGl0ZTItQ2l0eS5tbWRiKQoqIFtHZW9MaXRlMi1BU04ubW1kYl0oaHR0cHM6Ly9jcm93ZHNlYy1zdGF0aWNzLWFzc2V0cy5zMy1ldS13ZXN0LTEuYW1hem9uYXdzLmNvbS9HZW9MaXRlMi1BU04ubW1kYikKCg==", +- "content": "ZmlsdGVyOiAiJ3NvdXJjZV9pcCcgaW4gZXZ0Lk1ldGEiCm5hbWU6IGNyb3dkc2VjdXJpdHkvZ2VvaXAtZW5yaWNoCmRlc2NyaXB0aW9uOiAiUG9wdWxhdGUgZXZlbnQgd2l0aCBnZW9sb2MgaW5mbyA6IGFzLCBjb3VudHJ5LCBjb29yZHMsIHNvdXJjZSByYW5nZS4iCmRhdGE6CiAgLSBzb3VyY2VfdXJsOiBodHRwczovL2Nyb3dkc2VjLXN0YXRpY3MtYXNzZXRzLnMzLWV1LXdlc3QtMS5hbWF6b25hd3MuY29tL0dlb0xpdGUyLUNpdHkubW1kYgogICAgZGVzdF9maWxlOiBHZW9MaXRlMi1DaXR5Lm1tZGIKICAtIHNvdXJjZV91cmw6IGh0dHBzOi8vY3Jvd2RzZWMtc3RhdGljcy1hc3NldHMuczMtZXUtd2VzdC0xLmFtYXpvbmF3cy5jb20vR2VvTGl0ZTItQVNOLm1tZGIKICAgIGRlc3RfZmlsZTogR2VvTGl0ZTItQVNOLm1tZGIKc3RhdGljczoKICAtIG1ldGhvZDogR2VvSXBDaXR5CiAgICBleHByZXNzaW9uOiBldnQuTWV0YS5zb3VyY2VfaXAKICAtIG1ldGE6IElzb0NvZGUKICAgIGV4cHJlc3Npb246IGV2dC5FbnJpY2hlZC5Jc29Db2RlCiAgLSBtZXRhOiBJc0luRVUKICAgIGV4cHJlc3Npb246IGV2dC5FbnJpY2hlZC5Jc0luRVUKICAtIG1ldGE6IEdlb0Nvb3JkcwogICAgZXhwcmVzc2lvbjogZXZ0LkVucmljaGVkLkdlb0Nvb3JkcwogIC0gbWV0aG9kOiBHZW9JcEFTTgogICAgZXhwcmVzc2lvbjogZXZ0Lk1ldGEuc291cmNlX2lwCiAgLSBtZXRhOiBBU05OdW1iZXIKICAgIGV4cHJlc3Npb246IGV2dC5FbnJpY2hlZC5BU05OdW1iZXIKICAtIG1ldGE6IEFTTk9yZwogICAgZXhwcmVzc2lvbjogZXZ0LkVucmljaGVkLkFTTk9yZwogIC0gbWV0aG9kOiBJcFRvUmFuZ2UKICAgIGV4cHJlc3Npb246IGV2dC5NZXRhLnNvdXJjZV9pcAogIC0gbWV0YTogU291cmNlUmFuZ2UKICAgIGV4cHJlc3Npb246IGV2dC5FbnJpY2hlZC5Tb3VyY2VSYW5nZQo=", +- "description": "Populate event with geoloc info : as, country, coords, source range.", +- "author": "crowdsecurity", +- "labels": null +- }, + "crowdsecurity/haproxy-logs": { + "path": "parsers/s01-parse/crowdsecurity/haproxy-logs.yaml", + "stage": "s01-parse", +@@ -6375,4 +6351,4 @@ + } + } + } +-} +\ No newline at end of file ++} +--- a/hub1/collections/crowdsecurity/linux.yaml ++++ b/hub1/collections/crowdsecurity/linux.yaml +@@ -1,10 +1,9 @@ + parsers: + - crowdsecurity/syslog-logs +- - crowdsecurity/geoip-enrich + - crowdsecurity/dateparse-enrich + collections: + - crowdsecurity/sshd +-description: "core linux support : syslog+geoip+ssh" ++description: "core linux support : syslog+ssh" + author: crowdsecurity + tags: + - linux +--- a/hub1/parsers/s02-enrich/crowdsecurity/geoip-enrich.yaml ++++ /dev/null +@@ -1,27 +0,0 @@ +-filter: "'source_ip' in evt.Meta" +-name: crowdsecurity/geoip-enrich +-description: "Populate event with geoloc info : as, country, coords, source range." +-data: +- - source_url: https://crowdsec-statics-assets.s3-eu-west-1.amazonaws.com/GeoLite2-City.mmdb +- dest_file: GeoLite2-City.mmdb +- - source_url: https://crowdsec-statics-assets.s3-eu-west-1.amazonaws.com/GeoLite2-ASN.mmdb +- dest_file: GeoLite2-ASN.mmdb +-statics: +- - method: GeoIpCity +- expression: evt.Meta.source_ip +- - meta: IsoCode +- expression: evt.Enriched.IsoCode +- - meta: IsInEU +- expression: evt.Enriched.IsInEU +- - meta: GeoCoords +- expression: evt.Enriched.GeoCoords +- - method: GeoIpASN +- expression: evt.Meta.source_ip +- - meta: ASNNumber +- expression: evt.Enriched.ASNNumber +- - meta: ASNOrg +- expression: evt.Enriched.ASNOrg +- - method: IpToRange +- expression: evt.Meta.source_ip +- - meta: SourceRange +- expression: evt.Enriched.SourceRange diff --git a/patches/0005-adjust-config.patch b/patches/0005-adjust-config.patch new file mode 100644 index 0000000..acc3c2d --- /dev/null +++ b/patches/0005-adjust-config.patch @@ -0,0 +1,38 @@ +From: Cyril Brulebois +Date: Mon, 01 Mar 2021 14:11:36 +0000 +Last-Update: Tue, 29 Nov 2022 16:03:49 +0000 +Subject: Adjust default config + +Let's have all hub-related data under /var/lib/crowdsec/hub instead of +the default /etc/crowdsec/hub directory. + +Also fix plugin directory. + +Also delete pid_dir, which would otherwise generate this at runtime: + + Deprecation warning: the pid_dir config can be safely removed and is not required + +Signed-off-by: Cyril Brulebois +--- a/config/config.yaml ++++ b/config/config.yaml +@@ -1,6 +1,5 @@ + common: + daemonize: true +- pid_dir: /var/run/ + log_media: file + log_level: info + log_dir: /var/log/ +@@ -12,10 +11,10 @@ config_paths: + config_dir: /etc/crowdsec/ + data_dir: /var/lib/crowdsec/data/ + simulation_path: /etc/crowdsec/simulation.yaml +- hub_dir: /etc/crowdsec/hub/ +- index_path: /etc/crowdsec/hub/.index.json ++ hub_dir: /var/lib/crowdsec/hub/ ++ index_path: /var/lib/crowdsec/hub/.index.json + notification_dir: /etc/crowdsec/notifications/ +- plugin_dir: /usr/local/lib/crowdsec/plugins/ ++ plugin_dir: /usr/lib/crowdsec/plugins/ + crowdsec_service: + acquisition_path: /etc/crowdsec/acquis.yaml + acquisition_dir: /etc/crowdsec/acquis.d diff --git a/patches/0007-automatically-enable-online-hub.patch b/patches/0007-automatically-enable-online-hub.patch new file mode 100644 index 0000000..1a0e45a --- /dev/null +++ b/patches/0007-automatically-enable-online-hub.patch @@ -0,0 +1,38 @@ +From: Cyril Brulebois +Date: Mon, 01 Mar 2021 20:40:04 +0000 +Subject: Automatically enable the online hub + +By default, crowdsec comes with an offline copy of the hub (see +README.Debian). When running `cscli hub update`, ensure switching from +this offline copy to the online hub. + +To ensure cscli doesn't disable anything that was configured (due to +symlinks from /etc/crowdsec becoming dangling all of a sudden), copy the +offline hub in the live directory (/var/lib/crowdsec/hub), and let +further operations (`cscli hub upgrade`, or `cscli install`) +update the live directory as required. + +Signed-off-by: Cyril Brulebois +--- a/cmd/crowdsec-cli/hub.go ++++ b/cmd/crowdsec-cli/hub.go +@@ -2,6 +2,7 @@ package main + + import ( + "fmt" ++ "os/exec" + + "github.com/fatih/color" + log "github.com/sirupsen/logrus" +@@ -84,6 +85,12 @@ Fetches the [.index.json](https://github + return nil + }, + Run: func(cmd *cobra.Command, args []string) { ++ /* Make sure to move away from the offline hub (see README.Debian) */ ++ command := exec.Command("/var/lib/dpkg/info/crowdsec.postinst", "enable-online-hub") ++ if err := command.Run(); err != nil { ++ log.Printf("Enabling Online Hub failed with error: %v", err) ++ } ++ + if err := csConfig.LoadHub(); err != nil { + log.Fatal(err) + } diff --git a/patches/0008-r3labs-diff-versions.patch b/patches/0008-r3labs-diff-versions.patch new file mode 100644 index 0000000..d9e6bca --- /dev/null +++ b/patches/0008-r3labs-diff-versions.patch @@ -0,0 +1,22 @@ +--- a/go.mod ++++ b/go.mod +@@ -50,7 +50,7 @@ require ( + github.com/prometheus/client_golang v1.11.0 + github.com/prometheus/client_model v0.2.0 + github.com/prometheus/prom2json v1.3.0 +- github.com/r3labs/diff/v2 v2.14.1 ++ github.com/r3labs/diff/v3 v3.0.0 + github.com/sirupsen/logrus v1.8.1 + github.com/spf13/cobra v1.5.0 + github.com/stretchr/testify v1.8.0 +--- a/pkg/hubtest/parser_assert.go ++++ b/pkg/hubtest/parser_assert.go +@@ -17,7 +17,7 @@ import ( + "github.com/enescakir/emoji" + "github.com/fatih/color" + "github.com/pkg/errors" +- diff "github.com/r3labs/diff/v2" ++ diff "github.com/r3labs/diff/v3" + log "github.com/sirupsen/logrus" + "gopkg.in/yaml.v2" + ) diff --git a/patches/0009-disable-kafka-acquisition-module.patch b/patches/0009-disable-kafka-acquisition-module.patch new file mode 100644 index 0000000..f95f264 --- /dev/null +++ b/patches/0009-disable-kafka-acquisition-module.patch @@ -0,0 +1,37 @@ +--- a/pkg/acquisition/acquisition.go ++++ b/pkg/acquisition/acquisition.go +@@ -11,7 +11,6 @@ import ( + dockeracquisition "github.com/crowdsecurity/crowdsec/pkg/acquisition/modules/docker" + fileacquisition "github.com/crowdsecurity/crowdsec/pkg/acquisition/modules/file" + journalctlacquisition "github.com/crowdsecurity/crowdsec/pkg/acquisition/modules/journalctl" +- kafkaacquisition "github.com/crowdsecurity/crowdsec/pkg/acquisition/modules/kafka" + kinesisacquisition "github.com/crowdsecurity/crowdsec/pkg/acquisition/modules/kinesis" + syslogacquisition "github.com/crowdsecurity/crowdsec/pkg/acquisition/modules/syslog" + wineventlogacquisition "github.com/crowdsecurity/crowdsec/pkg/acquisition/modules/wineventlog" +@@ -71,10 +70,6 @@ var AcquisitionSources = []struct { + name: "wineventlog", + iface: func() DataSource { return &wineventlogacquisition.WinEventLogSource{} }, + }, +- { +- name: "kafka", +- iface: func() DataSource { return &kafkaacquisition.KafkaSource{} }, +- }, + } + + func GetDataSourceIface(dataSourceType string) DataSource { +--- a/pkg/acquisition/modules/kafka/kafka.go ++++ b/pkg/acquisition/modules/kafka/kafka.go +@@ -1,3 +1,5 @@ ++//go:build ignore ++ + package kafkaacquisition + + import ( +--- a/pkg/acquisition/modules/kafka/kafka_test.go ++++ b/pkg/acquisition/modules/kafka/kafka_test.go +@@ -1,3 +1,5 @@ ++//go:build ignore ++ + package kafkaacquisition + + import ( diff --git a/patches/0010-disable-some-tests.patch b/patches/0010-disable-some-tests.patch new file mode 100644 index 0000000..7a08f52 --- /dev/null +++ b/patches/0010-disable-some-tests.patch @@ -0,0 +1,24 @@ +--- a/pkg/acquisition/modules/cloudwatch/cloudwatch_test.go ++++ b/pkg/acquisition/modules/cloudwatch/cloudwatch_test.go +@@ -1,3 +1,5 @@ ++//go:build ignore ++ + package cloudwatchacquisition + + import ( +--- a/pkg/acquisition/modules/kinesis/kinesis_test.go ++++ b/pkg/acquisition/modules/kinesis/kinesis_test.go +@@ -1,3 +1,5 @@ ++//go:build ignore ++ + package kinesisacquisition + + import ( +--- a/pkg/acquisition/modules/docker/docker_test.go ++++ b/pkg/acquisition/modules/docker/docker_test.go +@@ -1,3 +1,5 @@ ++//go:build ignore ++ + package dockeracquisition + + import ( diff --git a/patches/0011-refresh-protobuf-code.patch b/patches/0011-refresh-protobuf-code.patch new file mode 100644 index 0000000..bcc91c1 --- /dev/null +++ b/patches/0011-refresh-protobuf-code.patch @@ -0,0 +1,428 @@ +From: Manuel Sabban +Date: Fri, 25 Nov 2022 08:34:47 +0100 +Subject: Refresh code generated from protobuf specifications + +Without this patch, crowdsec would build fine but would fail in its testsuite, +reaching the 10-minute timeout. This was tracked down to unexpected csplugins +crashes, due to mismatched protobuf versions. + +This issue would go away with golang-goprotobuf-dev 1.5.2-1~exp1 (as found in +experimental for a while) instead of 1.3.5-2 (in unstable as of November 2022). +Since protobuf's build-time and run-time are particularly tricky to get right, +incompatible 1.3.x and 1.5.x versions are packaged separately and aren't +co-installable. Since most packages depend on the 1.3.x version, it's not +possible to pull the 1.5.x version to avoid this issue. + +Therefore, upstream was kind enough to come up with this patch, refreshing the +generated code so that it works fine within unstable. + +Signed-off-by: Cyril Brulebois +--- a/pkg/protobufs/notifier.pb.go ++++ b/pkg/protobufs/notifier.pb.go +@@ -1,281 +1,170 @@ + // Code generated by protoc-gen-go. DO NOT EDIT. +-// versions: +-// protoc-gen-go v1.27.1 +-// protoc v3.12.4 + // source: notifier.proto + + package protobufs + + import ( + context "context" ++ fmt "fmt" ++ proto "github.com/golang/protobuf/proto" + grpc "google.golang.org/grpc" + codes "google.golang.org/grpc/codes" + status "google.golang.org/grpc/status" +- protoreflect "google.golang.org/protobuf/reflect/protoreflect" +- protoimpl "google.golang.org/protobuf/runtime/protoimpl" +- reflect "reflect" +- sync "sync" ++ math "math" + ) + +-const ( +- // Verify that this generated code is sufficiently up-to-date. +- _ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion) +- // Verify that runtime/protoimpl is sufficiently up-to-date. +- _ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20) +-) ++// Reference imports to suppress errors if they are not otherwise used. ++var _ = proto.Marshal ++var _ = fmt.Errorf ++var _ = math.Inf ++ ++// This is a compile-time assertion to ensure that this generated file ++// is compatible with the proto package it is being compiled against. ++// A compilation error at this line likely means your copy of the ++// proto package needs to be updated. ++const _ = proto.ProtoPackageIsVersion3 // please upgrade the proto package + + type Notification struct { +- state protoimpl.MessageState +- sizeCache protoimpl.SizeCache +- unknownFields protoimpl.UnknownFields +- +- Text string `protobuf:"bytes,1,opt,name=text,proto3" json:"text,omitempty"` +- Name string `protobuf:"bytes,2,opt,name=name,proto3" json:"name,omitempty"` ++ Text string `protobuf:"bytes,1,opt,name=text,proto3" json:"text,omitempty"` ++ Name string `protobuf:"bytes,2,opt,name=name,proto3" json:"name,omitempty"` ++ XXX_NoUnkeyedLiteral struct{} `json:"-"` ++ XXX_unrecognized []byte `json:"-"` ++ XXX_sizecache int32 `json:"-"` + } + +-func (x *Notification) Reset() { +- *x = Notification{} +- if protoimpl.UnsafeEnabled { +- mi := &file_notifier_proto_msgTypes[0] +- ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) +- ms.StoreMessageInfo(mi) +- } ++func (m *Notification) Reset() { *m = Notification{} } ++func (m *Notification) String() string { return proto.CompactTextString(m) } ++func (*Notification) ProtoMessage() {} ++func (*Notification) Descriptor() ([]byte, []int) { ++ return fileDescriptor_1c0fc606bc4470de, []int{0} + } + +-func (x *Notification) String() string { +- return protoimpl.X.MessageStringOf(x) ++func (m *Notification) XXX_Unmarshal(b []byte) error { ++ return xxx_messageInfo_Notification.Unmarshal(m, b) + } +- +-func (*Notification) ProtoMessage() {} +- +-func (x *Notification) ProtoReflect() protoreflect.Message { +- mi := &file_notifier_proto_msgTypes[0] +- if protoimpl.UnsafeEnabled && x != nil { +- ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) +- if ms.LoadMessageInfo() == nil { +- ms.StoreMessageInfo(mi) +- } +- return ms +- } +- return mi.MessageOf(x) ++func (m *Notification) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { ++ return xxx_messageInfo_Notification.Marshal(b, m, deterministic) + } +- +-// Deprecated: Use Notification.ProtoReflect.Descriptor instead. +-func (*Notification) Descriptor() ([]byte, []int) { +- return file_notifier_proto_rawDescGZIP(), []int{0} ++func (m *Notification) XXX_Merge(src proto.Message) { ++ xxx_messageInfo_Notification.Merge(m, src) ++} ++func (m *Notification) XXX_Size() int { ++ return xxx_messageInfo_Notification.Size(m) ++} ++func (m *Notification) XXX_DiscardUnknown() { ++ xxx_messageInfo_Notification.DiscardUnknown(m) + } + +-func (x *Notification) GetText() string { +- if x != nil { +- return x.Text ++var xxx_messageInfo_Notification proto.InternalMessageInfo ++ ++func (m *Notification) GetText() string { ++ if m != nil { ++ return m.Text + } + return "" + } + +-func (x *Notification) GetName() string { +- if x != nil { +- return x.Name ++func (m *Notification) GetName() string { ++ if m != nil { ++ return m.Name + } + return "" + } + + type Config struct { +- state protoimpl.MessageState +- sizeCache protoimpl.SizeCache +- unknownFields protoimpl.UnknownFields +- +- Config []byte `protobuf:"bytes,2,opt,name=config,proto3" json:"config,omitempty"` ++ Config []byte `protobuf:"bytes,2,opt,name=config,proto3" json:"config,omitempty"` ++ XXX_NoUnkeyedLiteral struct{} `json:"-"` ++ XXX_unrecognized []byte `json:"-"` ++ XXX_sizecache int32 `json:"-"` + } + +-func (x *Config) Reset() { +- *x = Config{} +- if protoimpl.UnsafeEnabled { +- mi := &file_notifier_proto_msgTypes[1] +- ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) +- ms.StoreMessageInfo(mi) +- } ++func (m *Config) Reset() { *m = Config{} } ++func (m *Config) String() string { return proto.CompactTextString(m) } ++func (*Config) ProtoMessage() {} ++func (*Config) Descriptor() ([]byte, []int) { ++ return fileDescriptor_1c0fc606bc4470de, []int{1} + } + +-func (x *Config) String() string { +- return protoimpl.X.MessageStringOf(x) ++func (m *Config) XXX_Unmarshal(b []byte) error { ++ return xxx_messageInfo_Config.Unmarshal(m, b) + } +- +-func (*Config) ProtoMessage() {} +- +-func (x *Config) ProtoReflect() protoreflect.Message { +- mi := &file_notifier_proto_msgTypes[1] +- if protoimpl.UnsafeEnabled && x != nil { +- ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) +- if ms.LoadMessageInfo() == nil { +- ms.StoreMessageInfo(mi) +- } +- return ms +- } +- return mi.MessageOf(x) ++func (m *Config) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { ++ return xxx_messageInfo_Config.Marshal(b, m, deterministic) + } +- +-// Deprecated: Use Config.ProtoReflect.Descriptor instead. +-func (*Config) Descriptor() ([]byte, []int) { +- return file_notifier_proto_rawDescGZIP(), []int{1} ++func (m *Config) XXX_Merge(src proto.Message) { ++ xxx_messageInfo_Config.Merge(m, src) ++} ++func (m *Config) XXX_Size() int { ++ return xxx_messageInfo_Config.Size(m) + } ++func (m *Config) XXX_DiscardUnknown() { ++ xxx_messageInfo_Config.DiscardUnknown(m) ++} ++ ++var xxx_messageInfo_Config proto.InternalMessageInfo + +-func (x *Config) GetConfig() []byte { +- if x != nil { +- return x.Config ++func (m *Config) GetConfig() []byte { ++ if m != nil { ++ return m.Config + } + return nil + } + + type Empty struct { +- state protoimpl.MessageState +- sizeCache protoimpl.SizeCache +- unknownFields protoimpl.UnknownFields ++ XXX_NoUnkeyedLiteral struct{} `json:"-"` ++ XXX_unrecognized []byte `json:"-"` ++ XXX_sizecache int32 `json:"-"` + } + +-func (x *Empty) Reset() { +- *x = Empty{} +- if protoimpl.UnsafeEnabled { +- mi := &file_notifier_proto_msgTypes[2] +- ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) +- ms.StoreMessageInfo(mi) +- } ++func (m *Empty) Reset() { *m = Empty{} } ++func (m *Empty) String() string { return proto.CompactTextString(m) } ++func (*Empty) ProtoMessage() {} ++func (*Empty) Descriptor() ([]byte, []int) { ++ return fileDescriptor_1c0fc606bc4470de, []int{2} + } + +-func (x *Empty) String() string { +- return protoimpl.X.MessageStringOf(x) ++func (m *Empty) XXX_Unmarshal(b []byte) error { ++ return xxx_messageInfo_Empty.Unmarshal(m, b) ++} ++func (m *Empty) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { ++ return xxx_messageInfo_Empty.Marshal(b, m, deterministic) ++} ++func (m *Empty) XXX_Merge(src proto.Message) { ++ xxx_messageInfo_Empty.Merge(m, src) ++} ++func (m *Empty) XXX_Size() int { ++ return xxx_messageInfo_Empty.Size(m) ++} ++func (m *Empty) XXX_DiscardUnknown() { ++ xxx_messageInfo_Empty.DiscardUnknown(m) + } + +-func (*Empty) ProtoMessage() {} ++var xxx_messageInfo_Empty proto.InternalMessageInfo + +-func (x *Empty) ProtoReflect() protoreflect.Message { +- mi := &file_notifier_proto_msgTypes[2] +- if protoimpl.UnsafeEnabled && x != nil { +- ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) +- if ms.LoadMessageInfo() == nil { +- ms.StoreMessageInfo(mi) +- } +- return ms +- } +- return mi.MessageOf(x) ++func init() { ++ proto.RegisterType((*Notification)(nil), "proto.Notification") ++ proto.RegisterType((*Config)(nil), "proto.Config") ++ proto.RegisterType((*Empty)(nil), "proto.Empty") + } + +-// Deprecated: Use Empty.ProtoReflect.Descriptor instead. +-func (*Empty) Descriptor() ([]byte, []int) { +- return file_notifier_proto_rawDescGZIP(), []int{2} ++func init() { ++ proto.RegisterFile("notifier.proto", fileDescriptor_1c0fc606bc4470de) + } + +-var File_notifier_proto protoreflect.FileDescriptor +- +-var file_notifier_proto_rawDesc = []byte{ +- 0x0a, 0x0e, 0x6e, 0x6f, 0x74, 0x69, 0x66, 0x69, 0x65, 0x72, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, +- 0x12, 0x05, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x22, 0x36, 0x0a, 0x0c, 0x4e, 0x6f, 0x74, 0x69, 0x66, +- 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x12, 0x12, 0x0a, 0x04, 0x74, 0x65, 0x78, 0x74, 0x18, +- 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x74, 0x65, 0x78, 0x74, 0x12, 0x12, 0x0a, 0x04, 0x6e, +- 0x61, 0x6d, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x22, +- 0x20, 0x0a, 0x06, 0x43, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x12, 0x16, 0x0a, 0x06, 0x63, 0x6f, 0x6e, +- 0x66, 0x69, 0x67, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x06, 0x63, 0x6f, 0x6e, 0x66, 0x69, +- 0x67, 0x22, 0x07, 0x0a, 0x05, 0x45, 0x6d, 0x70, 0x74, 0x79, 0x32, 0x61, 0x0a, 0x08, 0x4e, 0x6f, +- 0x74, 0x69, 0x66, 0x69, 0x65, 0x72, 0x12, 0x2b, 0x0a, 0x06, 0x4e, 0x6f, 0x74, 0x69, 0x66, 0x79, +- 0x12, 0x13, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x2e, 0x4e, 0x6f, 0x74, 0x69, 0x66, 0x69, 0x63, +- 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x1a, 0x0c, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x2e, 0x45, 0x6d, +- 0x70, 0x74, 0x79, 0x12, 0x28, 0x0a, 0x09, 0x43, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x75, 0x72, 0x65, +- 0x12, 0x0d, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x2e, 0x43, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x1a, +- 0x0c, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x2e, 0x45, 0x6d, 0x70, 0x74, 0x79, 0x42, 0x0d, 0x5a, +- 0x0b, 0x2e, 0x3b, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x73, 0x62, 0x06, 0x70, 0x72, +- 0x6f, 0x74, 0x6f, 0x33, +-} +- +-var ( +- file_notifier_proto_rawDescOnce sync.Once +- file_notifier_proto_rawDescData = file_notifier_proto_rawDesc +-) +- +-func file_notifier_proto_rawDescGZIP() []byte { +- file_notifier_proto_rawDescOnce.Do(func() { +- file_notifier_proto_rawDescData = protoimpl.X.CompressGZIP(file_notifier_proto_rawDescData) +- }) +- return file_notifier_proto_rawDescData +-} +- +-var file_notifier_proto_msgTypes = make([]protoimpl.MessageInfo, 3) +-var file_notifier_proto_goTypes = []interface{}{ +- (*Notification)(nil), // 0: proto.Notification +- (*Config)(nil), // 1: proto.Config +- (*Empty)(nil), // 2: proto.Empty +-} +-var file_notifier_proto_depIdxs = []int32{ +- 0, // 0: proto.Notifier.Notify:input_type -> proto.Notification +- 1, // 1: proto.Notifier.Configure:input_type -> proto.Config +- 2, // 2: proto.Notifier.Notify:output_type -> proto.Empty +- 2, // 3: proto.Notifier.Configure:output_type -> proto.Empty +- 2, // [2:4] is the sub-list for method output_type +- 0, // [0:2] is the sub-list for method input_type +- 0, // [0:0] is the sub-list for extension type_name +- 0, // [0:0] is the sub-list for extension extendee +- 0, // [0:0] is the sub-list for field type_name +-} +- +-func init() { file_notifier_proto_init() } +-func file_notifier_proto_init() { +- if File_notifier_proto != nil { +- return +- } +- if !protoimpl.UnsafeEnabled { +- file_notifier_proto_msgTypes[0].Exporter = func(v interface{}, i int) interface{} { +- switch v := v.(*Notification); i { +- case 0: +- return &v.state +- case 1: +- return &v.sizeCache +- case 2: +- return &v.unknownFields +- default: +- return nil +- } +- } +- file_notifier_proto_msgTypes[1].Exporter = func(v interface{}, i int) interface{} { +- switch v := v.(*Config); i { +- case 0: +- return &v.state +- case 1: +- return &v.sizeCache +- case 2: +- return &v.unknownFields +- default: +- return nil +- } +- } +- file_notifier_proto_msgTypes[2].Exporter = func(v interface{}, i int) interface{} { +- switch v := v.(*Empty); i { +- case 0: +- return &v.state +- case 1: +- return &v.sizeCache +- case 2: +- return &v.unknownFields +- default: +- return nil +- } +- } +- } +- type x struct{} +- out := protoimpl.TypeBuilder{ +- File: protoimpl.DescBuilder{ +- GoPackagePath: reflect.TypeOf(x{}).PkgPath(), +- RawDescriptor: file_notifier_proto_rawDesc, +- NumEnums: 0, +- NumMessages: 3, +- NumExtensions: 0, +- NumServices: 1, +- }, +- GoTypes: file_notifier_proto_goTypes, +- DependencyIndexes: file_notifier_proto_depIdxs, +- MessageInfos: file_notifier_proto_msgTypes, +- }.Build() +- File_notifier_proto = out.File +- file_notifier_proto_rawDesc = nil +- file_notifier_proto_goTypes = nil +- file_notifier_proto_depIdxs = nil ++var fileDescriptor_1c0fc606bc4470de = []byte{ ++ // 177 bytes of a gzipped FileDescriptorProto ++ 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xe2, 0xe2, 0xcb, 0xcb, 0x2f, 0xc9, ++ 0x4c, 0xcb, 0x4c, 0x2d, 0xd2, 0x2b, 0x28, 0xca, 0x2f, 0xc9, 0x17, 0x62, 0x05, 0x53, 0x4a, 0x66, ++ 0x5c, 0x3c, 0x7e, 0x60, 0x89, 0xe4, 0xc4, 0x92, 0xcc, 0xfc, 0x3c, 0x21, 0x21, 0x2e, 0x96, 0x92, ++ 0xd4, 0x8a, 0x12, 0x09, 0x46, 0x05, 0x46, 0x0d, 0xce, 0x20, 0x30, 0x1b, 0x24, 0x96, 0x97, 0x98, ++ 0x9b, 0x2a, 0xc1, 0x04, 0x11, 0x03, 0xb1, 0x95, 0x14, 0xb8, 0xd8, 0x9c, 0xf3, 0xf3, 0xd2, 0x32, ++ 0xd3, 0x85, 0xc4, 0xb8, 0xd8, 0x92, 0xc1, 0x2c, 0xb0, 0x3c, 0x4f, 0x10, 0x94, 0xa7, 0xc4, 0xce, ++ 0xc5, 0xea, 0x9a, 0x5b, 0x50, 0x52, 0x69, 0x94, 0xc8, 0xc5, 0xe1, 0x07, 0xb5, 0x5b, 0x48, 0x9b, ++ 0x8b, 0x0d, 0xcc, 0xae, 0x14, 0x12, 0x86, 0xb8, 0x43, 0x0f, 0xd9, 0x76, 0x29, 0x1e, 0xa8, 0x20, ++ 0x58, 0xa3, 0x90, 0x06, 0x17, 0x27, 0xc4, 0x8e, 0xd2, 0xa2, 0x54, 0x21, 0x5e, 0xa8, 0x14, 0x44, ++ 0x04, 0x55, 0xa5, 0x13, 0x6f, 0x14, 0xb7, 0x9e, 0x35, 0x58, 0x20, 0xa9, 0x34, 0xad, 0x38, 0x89, ++ 0x0d, 0xcc, 0x34, 0x06, 0x04, 0x00, 0x00, 0xff, 0xff, 0x7e, 0x86, 0x13, 0x58, 0xf4, 0x00, 0x00, ++ 0x00, + } + + // Reference imports to suppress errors if they are not otherwise used. +@@ -330,10 +219,10 @@ type NotifierServer interface { + type UnimplementedNotifierServer struct { + } + +-func (*UnimplementedNotifierServer) Notify(context.Context, *Notification) (*Empty, error) { ++func (*UnimplementedNotifierServer) Notify(ctx context.Context, req *Notification) (*Empty, error) { + return nil, status.Errorf(codes.Unimplemented, "method Notify not implemented") + } +-func (*UnimplementedNotifierServer) Configure(context.Context, *Config) (*Empty, error) { ++func (*UnimplementedNotifierServer) Configure(ctx context.Context, req *Config) (*Empty, error) { + return nil, status.Errorf(codes.Unimplemented, "method Configure not implemented") + } + diff --git a/patches/0013-skip-flakky-tests.patch b/patches/0013-skip-flakky-tests.patch new file mode 100644 index 0000000..5946587 --- /dev/null +++ b/patches/0013-skip-flakky-tests.patch @@ -0,0 +1,75 @@ +From: Cyril Brulebois +Date: Wed, 01 Mar 2023 22:41:36 +0100 +Subject: Skip flakky tests + +Initially (2023-02-15): + - TestAPICSendMetrics + - TestLongRunningQPS + +Both aim at checking performance, and might error out when the underlying +system is “slow”. That was the case for the first test on arm64 (KO on +arm-arm-01, OK on arm-ubc-03) and on armel (KO on arm-ubc-06, OK on +arm-conova-02), and for the second test on mipsel (KO on mipsel-aql-02). + +The same might happen on ci.debian.net, so disable both tests everywhere. + +Extension (2023-03-01): + - TestAPICCAPIPullIsOld + - TestAPICHandleDeletedDecisions + - TestAPICPullTop + - TestGetDecisionsSinceCount + +--- a/pkg/apiserver/apic_test.go ++++ b/pkg/apiserver/apic_test.go +@@ -95,6 +95,7 @@ func assertTotalAlertCount(t *testing.T, + } + + func TestAPICCAPIPullIsOld(t *testing.T) { ++ t.Skip() + api := getAPIC(t) + + isOld, err := api.CAPIPullIsOld() +@@ -223,6 +224,7 @@ func TestNewAPIC(t *testing.T) { + } + + func TestAPICHandleDeletedDecisions(t *testing.T) { ++ t.Skip() + api := getAPIC(t) + _, deleteCounters := makeAddAndDeleteCounters() + +@@ -502,6 +504,7 @@ func TestFillAlertsWithDecisions(t *test + } + + func TestAPICPullTop(t *testing.T) { ++ t.Skip() + api := getAPIC(t) + api.dbClient.Ent.Decision.Create(). + SetOrigin(SCOPE_LISTS). +@@ -699,6 +702,7 @@ func TestAPICPush(t *testing.T) { + } + + func TestAPICSendMetrics(t *testing.T) { ++ t.Skip() + tests := []struct { + name string + duration time.Duration +--- a/pkg/time/rate/rate_test.go ++++ b/pkg/time/rate/rate_test.go +@@ -175,6 +175,7 @@ func TestSimultaneousRequests(t *testing + } + + func TestLongRunningQPS(t *testing.T) { ++ t.Skip() + if testing.Short() { + t.Skip("skipping in short mode") + } +--- a/pkg/exprhelpers/exprlib_test.go ++++ b/pkg/exprhelpers/exprlib_test.go +@@ -847,6 +847,7 @@ func TestGetDecisionsCount(t *testing.T) + } + } + func TestGetDecisionsSinceCount(t *testing.T) { ++ t.Skip() + var err error + var start_ip, start_sfx, end_ip, end_sfx int64 + var ip_sz int diff --git a/patches/0014-silence-yaml-patching.patch b/patches/0014-silence-yaml-patching.patch new file mode 100644 index 0000000..7b3fc3e --- /dev/null +++ b/patches/0014-silence-yaml-patching.patch @@ -0,0 +1,33 @@ +From: Cyril Brulebois +Date: Wed, 01 Mar 2023 22:36:00 +0100 +Subject: Silence yaml patching + +Being able to patch configuration files using .local snippets is relatively new, +and very useful when it comes to shipping mostly untouched configuration files, +with just a few values set in a .local file. + +Unfortunately the initial code is very chatty and generates info-level messages +in all cscli calls. + +An early patch upstream is ca12432a2acd2fd607e9fbea97fea3fb3f124678 (backported +here), which demotes those messages from info to debug. It left the “Prepending” +messages at info, but those haven't been seen with the current packages, so +that's left untouched in this patch as well. + +Further refinement landed in cd4dabde0ec833552881dd36780ab847cf20882d but that +touches more code, and we're closing in freeze-wise, so leave it alone. All our +crowdsec* packages ship a README.Debian file, which should be sufficient in most +cases. When in doubt, adjusting the log level in specific commands should be +enough to figure out what's happening. + +--- a/pkg/yamlpatch/patcher.go ++++ b/pkg/yamlpatch/patcher.go +@@ -60,7 +60,7 @@ func (p *Patcher) MergedPatchContent() ( + return nil, err + } + if err == nil { +- log.Infof("Patching yaml: '%s' with '%s'", p.BaseFilePath, p.PatchFilePath) ++ log.Debugf("Patching yaml: '%s' with '%s'", p.BaseFilePath, p.PatchFilePath) + } + + var patched *bytes.Buffer diff --git a/patches/0015-silence-not-latest-version.patch b/patches/0015-silence-not-latest-version.patch new file mode 100644 index 0000000..4773361 --- /dev/null +++ b/patches/0015-silence-not-latest-version.patch @@ -0,0 +1,23 @@ +From: Cyril Brulebois +Date: Wed, 01 Mar 2023 23:58:00 +0100 +Subject: Silence “Crowdsec is not the latest version” messages. + +We're shipping crowdsec in a stable Debian release, which isn't quite +compatible with a “latest and greatest” approach. :) + +Upstream maintains a hub branch for the version we ship in stable anyway. + +--- a/pkg/cwhub/helpers.go ++++ b/pkg/cwhub/helpers.go +@@ -35,11 +35,6 @@ func chooseHubBranch() (string, error) { + return "master", nil + } + +- log.Warnf("Crowdsec is not the latest version. "+ +- "Current version is '%s' and the latest stable version is '%s'. Please update it!", +- csVersion, latest) +- log.Warnf("As a result, you will not be able to use parsers/scenarios/collections "+ +- "added to Crowdsec Hub after CrowdSec %s", latest) + return csVersion, nil + } + diff --git a/patches/0016-try-to-make-reproducible-build-work-2119.patch b/patches/0016-try-to-make-reproducible-build-work-2119.patch new file mode 100644 index 0000000..0a3121c --- /dev/null +++ b/patches/0016-try-to-make-reproducible-build-work-2119.patch @@ -0,0 +1,51 @@ +From b451d190b766d7fb5e64c7f1cedde314b02e1f0c Mon Sep 17 00:00:00 2001 +From: Manuel Sabban +Date: Mon, 13 Mar 2023 17:26:33 +0100 +Subject: [PATCH] try to make reproducible build work (#2119) + +Co-authored-by: sabban <15465465+sabban@users.noreply.github.com> +--- + .../modules/syslog/internal/parser/rfc3164/parse_test.go | 6 ++++-- + 1 file changed, 4 insertions(+), 2 deletions(-) + +diff --git a/pkg/acquisition/modules/syslog/internal/parser/rfc3164/parse_test.go b/pkg/acquisition/modules/syslog/internal/parser/rfc3164/parse_test.go +index bef3bde3..48772d59 100644 +--- a/pkg/acquisition/modules/syslog/internal/parser/rfc3164/parse_test.go ++++ b/pkg/acquisition/modules/syslog/internal/parser/rfc3164/parse_test.go +@@ -1,6 +1,7 @@ + package rfc3164 + + import ( ++ "fmt" + "testing" + "time" + ) +@@ -47,6 +48,7 @@ func TestPri(t *testing.T) { + } + + func TestTimestamp(t *testing.T) { ++ + tests := []struct { + input string + expected string +@@ -54,7 +56,7 @@ func TestTimestamp(t *testing.T) { + currentYear bool + }{ + {"May 20 09:33:54", "0000-05-20T09:33:54Z", "", false}, +- {"May 20 09:33:54", "2023-05-20T09:33:54Z", "", true}, ++ {"May 20 09:33:54", fmt.Sprintf("%d-05-20T09:33:54Z", time.Now().Year()), "", true}, + {"May 20 09:33:54 2022", "2022-05-20T09:33:54Z", "", false}, + {"May 1 09:33:54 2022", "2022-05-01T09:33:54Z", "", false}, + {"May 01 09:33:54 2021", "2021-05-01T09:33:54Z", "", true}, +@@ -257,7 +259,7 @@ func TestParse(t *testing.T) { + }, + { + "<12>May 20 09:33:54 UDMPRO,a2edd0c6ae48,udm-1.10.0.3686 kernel: foo", expected{ +- Timestamp: time.Date(2023, time.May, 20, 9, 33, 54, 0, time.UTC), ++ Timestamp: time.Date(time.Now().Year(), time.May, 20, 9, 33, 54, 0, time.UTC), + Hostname: "UDMPRO,a2edd0c6ae48,udm-1.10.0.3686", + Tag: "kernel", + PID: "", +-- +2.30.2 + diff --git a/patches/series b/patches/series new file mode 100644 index 0000000..ad42cd4 --- /dev/null +++ b/patches/series @@ -0,0 +1,12 @@ +0003-adjust-systemd-unit.patch +0004-disable-geoip-enrich.patch +0005-adjust-config.patch +0007-automatically-enable-online-hub.patch +0008-r3labs-diff-versions.patch +0009-disable-kafka-acquisition-module.patch +0010-disable-some-tests.patch +0011-refresh-protobuf-code.patch +0013-skip-flakky-tests.patch +0014-silence-yaml-patching.patch +0015-silence-not-latest-version.patch +0016-try-to-make-reproducible-build-work-2119.patch diff --git a/rules b/rules new file mode 100755 index 0000000..fb37cc4 --- /dev/null +++ b/rules @@ -0,0 +1,77 @@ +#!/usr/bin/make -f + +export DH_GOLANG_INSTALL_ALL := 1 +export DH_GOLANG_EXCLUDES := hub\d+ data\d+ + +# Make sure ancillary rules don't generate errors, since the following +# variable is only set while building: +ifneq (,$(SOURCE_DATE_EPOCH)) + now := @$(SOURCE_DATE_EPOCH) +else + now := now +endif + +# Upstream's Makefile would take care of this, but we're building via +# dh_auto_build in golang mode, instead of using make: +export BUILD_VERSION := v$(shell dpkg-parsechangelog -SVersion) +export BUILD_TAG := debian +export BUILD_CODENAME := $(shell perl -ne 'print $$1 if /^BUILD_CODENAME="(.+)"$$/' platform/unix_common.mk) +export BUILD_TIMESTAMP := $(shell TZ=Etc/UTC date +'%F_%T' -d $(now)) +export set_cwversion := -X github.com/crowdsecurity/crowdsec/pkg/cwversion +export LD_FLAGS := -ldflags '-s -w \ + $(set_cwversion).Version=$(BUILD_VERSION) \ + $(set_cwversion).Tag=$(BUILD_TAG) \ + $(set_cwversion).Codename=$(BUILD_CODENAME) \ + $(set_cwversion).BuildDate=$(BUILD_TIMESTAMP) \ +' + +# Use 1 for a new upstream release, and bump it when an update of the +# hub files is desired while the upstream version doesn't change. See +# below for the generate_hub_tarball target: +export DATA_ID := 1 +export HUB_ID := 1 +export HUB_BRANCH := origin/v1.4.6 +export HUB_DIR := ../hub +export U_VERSION := $(shell dpkg-parsechangelog -SVersion|sed 's/-.*//') + +%: + dh $@ --builddirectory=_build --buildsystem=golang --with=golang + +override_dh_auto_build: + dh_auto_build -- $(LD_FLAGS) + +override_dh_install-arch: + dh_install -a + # Switch from Golang naming to upstream-desired naming: + mv debian/crowdsec/usr/bin/crowdsec-cli \ + debian/crowdsec/usr/bin/cscli + # Adjust the hub branch according to the upstream version: + sed "s/\(.*hub_branch:\) master/\1 v$(U_VERSION)/" -i debian/crowdsec/etc/crowdsec/config.yaml + # Drop unit tests from the hub: + find debian/crowdsec/usr/share/crowdsec/hub -depth -name '.tests' -exec rm -rf '{}' ';' + +# #1031328 (upstream #2125): +override_dh_install-indep: IDIR=debian/golang-github-crowdsecurity-crowdsec-dev/usr/share/gocode/src/github.com/crowdsecurity/crowdsec/pkg/cwhub +override_dh_install-indep: + dh_install -i + rm -f $(IDIR)/hubdir/.index.json + rm -f $(IDIR)/hubdir/collections/crowdsecurity/test_collection.yaml + rm -f $(IDIR)/hubdir/scenarios/crowdsecurity/barfoo_scenario.yaml + rm -f $(IDIR)/hubdir/scenarios/crowdsecurity/foobar_scenario.yaml + rm -f $(IDIR)/hubdir/collections/crowdsecurity/test_collection.yaml + rm -f $(IDIR)/hubdir/scenarios/crowdsecurity/barfoo_scenario.yaml + rm -f $(IDIR)/install/collections/test_collection.yaml + rm -f $(IDIR)/install/scenarios/barfoo_scenario.yaml + + +### Maintainer targets: + +generate_hub_tarball: + cd $(HUB_DIR) && git archive --prefix hub$(HUB_ID)/ $(HUB_BRANCH) | gzip -9 > ../crowdsec_$(U_VERSION).orig-hub$(HUB_ID).tar.gz \ + && echo "Generated hub tarball from branch $(HUB_BRANCH), at commit `git show $(HUB_BRANCH) | awk '/^commit / {print $$2; quit}' | cut -b -10`" + +extract_hub_tarball: + tar xf ../crowdsec_$(U_VERSION).orig-hub$(HUB_ID).tar.gz + +extract_data_tarball: + tar xf ../crowdsec_$(U_VERSION).orig-data$(HUB_ID).tar.gz diff --git a/source/format b/source/format new file mode 100644 index 0000000..163aaf8 --- /dev/null +++ b/source/format @@ -0,0 +1 @@ +3.0 (quilt) diff --git a/upstream/metadata b/upstream/metadata new file mode 100644 index 0000000..0038940 --- /dev/null +++ b/upstream/metadata @@ -0,0 +1,5 @@ +--- +Bug-Database: https://github.com/crowdsecurity/crowdsec/issues +Bug-Submit: https://github.com/crowdsecurity/crowdsec/issues/new +Repository: https://github.com/crowdsecurity/crowdsec.git +Repository-Browse: https://github.com/crowdsecurity/crowdsec diff --git a/watch b/watch new file mode 100644 index 0000000..f6c12be --- /dev/null +++ b/watch @@ -0,0 +1,4 @@ +version=4 +opts="filenamemangle=s%(?:.*?)?v?(\d[\d.]*)\.tar\.gz%crowdsec-$1.tar.gz%,\ + uversionmangle=s/(\d)[_\.\-\+]?(RC|rc|pre|dev|beta|alpha)[.]?(\d*)$/\$1~\$2\$3/" \ + https://github.com/crowdsecurity/crowdsec/tags .*/v?(\d\S*)\.tar\.gz debian