Commit Graph

939 Commits

Author SHA1 Message Date
Donncha Ó Cearbhaill
a2386dbdf7 Refactor Android backup password handling and add tests 2023-07-22 19:17:27 +02:00
Donncha Ó Cearbhaill
019cfbb84e
Merge pull request #363 from aticu/main
Add option to disable interactivity and pass Android backup password on CLI
2023-07-22 16:44:35 +02:00
Donncha Ó Cearbhaill
3d924e22ec Merge branch 'release/v2.4.0' 2023-07-21 12:17:32 +02:00
Donncha Ó Cearbhaill
ca3c1bade4 Bump version to v2.4.0
Bumping the minor version as we introduce some backwards-incompatible
API changes to module definition in #367.
2023-07-21 12:14:31 +02:00
Donncha Ó Cearbhaill
85877fd3eb
Merge pull request #369 from mvt-project/move-indicator-checking
Move detection and alerts from run() to check_indicators()
2023-07-21 12:12:36 +02:00
Donncha Ó Cearbhaill
8015ff78e8 Fix black error 2023-07-21 12:10:45 +02:00
Donncha Ó Cearbhaill
1a07b9a78f Move syntax checking before unit tests 2023-07-21 11:30:59 +02:00
Donncha Ó Cearbhaill
0b88de9867 Move detection and alerts from run() to check_indicators() 2023-07-21 11:29:12 +02:00
Niclas Schwarzlose
0edc9d7b81 Add option to disable interactivity 2023-07-19 11:29:51 +02:00
Donncha Ó Cearbhaill
76d7534b05 Fix bug recording detections in WebkitResourceLoadStatistics module 2023-07-18 18:02:42 +02:00
Donncha Ó Cearbhaill
ae2ab02347
Merge pull request #367 from mvt-project/refactor-module-options
Add a module_options parameter to pass data from CLI to modules
2023-07-17 19:07:41 +02:00
Donncha Ó Cearbhaill
e2c623c40f Move --fast flag from being a top-level MVT module parameter to an option in a new module_options parameter 2023-07-17 18:52:35 +02:00
Christian Clauss
a6e1a3de12
Add GitHub Annotions to ruff output (#364)
* Add GitHub Annotions to ruff output
* Upgrade GitHub Actions
* No Py3.11
2023-07-15 14:42:13 +02:00
tek
e7270d6a07 Fixes import and adds test for PR 361 2023-07-10 22:55:22 +02:00
Niclas Schwarzlose
1968a0fca2
Improve appops parsing in dumpsys (#361)
Without this change the package doesn't get properly reset when a new
user starts.

See for example in this excerpt:

```
 1 |    Package com.android.bluetooth:
 2 |      READ_CONTACTS (allow):
 3 |        null=[
 4 |          Access: [pers-s] 2022-04-22 13:24:17.577 (-277d5h22m53s447ms)
 5 |        ]
 6 |      WAKE_LOCK (allow):
 7 |        null=[
 8 |          Access: [pers-s] 2023-01-24 17:45:49.712 (-1m21s312ms) duration=+3ms
 9 |        ]
10 |      GET_USAGE_STATS (default):
11 |        null=[
12 |          Reject: [pers-s]2022-04-22 13:23:53.964 (-277d5h23m17s60ms)
13 |        ]
14 |      BLUETOOTH_CONNECT (allow):
15 |        null=[
16 |          Access: [pers-s] 2022-04-22 13:23:53.988 (-277d5h23m17s36ms)
17 |        ]
18 |  Uid 1027:
19 |    state=pers
20 |    capability=LCMN
21 |    appWidgetVisible=false
22 |      LEGACY_STORAGE: mode=ignore
23 |    Package com.android.nfc:
24 |      WAKE_LOCK (allow):
25 |        null=[
26 |          Access: [pers-s] 2022-04-22 13:23:54.633 (-277d5h23m16s391ms) duration=+1s73ms
27 |        ]
```

Here the package "com.android.bluetooth" is not reset when in line 18,
so when "LEGACY_STORAGE:" in line 22 is encountered, it's added as
another permission to "com.android.bluetooth" with "access" set to
"ode=igno".

This PR fixes that by resetting the package whenever a new Uid is
encountered.

Co-authored-by: Niclas Schwarzlose <niclas.schwarzlose@reporter-ohne-grenzen.de>
2023-07-10 22:53:58 +02:00
Donncha Ó Cearbhaill
46cc54df74 Add information about public indicators and support avenues to documentation 2023-06-30 19:43:30 +02:00
Donncha Ó Cearbhaill
7046ff80d1 Add SMS read time in the MVT logs 2023-06-30 19:30:50 +02:00
Donncha Ó Cearbhaill
e2516f284b
Bump version number 2023-06-29 17:03:26 +02:00
Donncha Ó Cearbhaill
17963f83d6 Fix URL to indicator repo in docs 2023-06-29 16:49:20 +02:00
Donncha Ó Cearbhaill
4f0c9c6077 Update README with information on indicators of compromise and path ways for forensic support 2023-06-29 16:48:56 +02:00
Donncha Ó Cearbhaill
27bd5f03a8
Merge pull request #359 from mvt-project/optimise-domain-checking
Optimise domain checking performance
2023-06-29 14:56:50 +02:00
Donncha Ó Cearbhaill
3babbadc1d Add docs for the profiling feature 2023-06-29 14:55:09 +02:00
Donncha Ó Cearbhaill
41db117168 Improve performance when checking URLs and domains
Some MVT modules such as the WhatsApp module can be very slow as it was taking a naive approach to look for IOCs. The code was checking URLs (potentially more than 100k) against
1000's of IOC domains resulting in a quadratic run-time with hundreds of millions of comparisons as the number of IOCs increases.

This commit add an Aho-Corasick library which allows the efficient search in a string (the URL in this case) for all matches in set of keys (the IOCs). This data structure is perfect for this use case.

A quick measurement shows a 80% performance improvement for a WhatsApp database with 100k entries. The slow path is now the time spent fetching and expanding short URLs found in the database. This
can also be sped up significantly by fetching each URL asynchronously. This would require reworking modules to split the URL expansion from the IOC check so I will implement in a separate PR.
2023-06-29 14:14:44 +02:00
Donncha Ó Cearbhaill
2b01ed7179 Add optional profiling for MVT modules 2023-06-29 13:31:13 +02:00
Donncha Ó Cearbhaill
78d493b17e
Merge pull request #356 from mvt-project/auto/add-new-ios-releases
[auto] Update iOS releases and versions
2023-06-22 11:06:45 +02:00
DonnchaC
473c80009b Add new iOS versions and build numbers 2023-06-22 00:17:52 +00:00
tek
a1481683e3 Adapts linter workflow to black 2023-06-14 01:05:14 +02:00
Nex
bdd36a9179
Merge pull request #349 from mvt-project/code-cleanup
Linted code using isort + autoflake + black
2023-06-08 21:12:34 +02:00
Nex
e1677639c4 Linted code using isort + autoflake + black, fixed wrong use of Optional[bool] 2023-06-01 23:40:26 +02:00
tek
c2d740ed36 Handle better some empty database issues in iOS backups 2023-05-25 00:24:34 +02:00
tek
d0e24c6369 Fixes a bug in the applications module 2023-05-24 12:04:03 +02:00
tek
a1994079b1 Sort imports 2023-05-24 12:03:49 +02:00
Donncha Ó Cearbhaill
289b7efdeb Add missing iOS build numbers 2023-05-21 17:11:07 +01:00
Donncha Ó Cearbhaill
166a63e14c
Merge pull request #347 from mvt-project/auto/add-new-ios-releases
[auto] Update iOS releases and versions
2023-05-21 17:54:25 +02:00
DonnchaC
1b933fdb12 Add new iOS versions and build numbers 2023-05-21 15:53:45 +00:00
Donncha Ó Cearbhaill
0c0ff7012b Set branch number for auto-generated pull request 2023-05-21 16:52:47 +01:00
Donncha Ó Cearbhaill
f9b0d07a81 Don't include information beta's in the version JSON 2023-05-21 16:49:14 +01:00
Donncha Ó Cearbhaill
d14bcdd05f Update title used in auto PR for new iOS versions 2023-05-21 16:47:56 +01:00
Donncha Ó Cearbhaill
e026bb0a76 Fix path to script in workflow 2023-05-21 16:44:17 +01:00
Donncha Ó Cearbhaill
253b4f031a Allow workflow to be triggered manually 2023-05-21 16:42:54 +01:00
Donncha Ó Cearbhaill
ec14297643
Merge pull request #345 from mvt-project/feature/auto-update-version-info
Add workflow to auto-update iOS builds and version numbers
2023-05-21 17:38:46 +02:00
Donncha Ó Cearbhaill
3142d86edd Fix path to include version JSON files in built package 2023-05-21 16:37:36 +01:00
Donncha Ó Cearbhaill
c18998d771 Add version 16.5 to resolve merge conflict from main 2023-05-21 16:26:12 +01:00
Donncha Ó Cearbhaill
22fd794fb8 Fix python style and setup.cfg syntax 2023-05-21 16:15:49 +01:00
Donncha Ó Cearbhaill
27c5c76dc2 Add script and worker to auto-update build and version info 2023-05-21 16:09:50 +01:00
Donncha Ó Cearbhaill
fafbac3545 Fix sorting of version numbers 2023-05-20 21:49:27 +01:00
Donncha Ó Cearbhaill
bbfaadd297 Load iOS device and build information from a JSON file. 2023-05-20 21:24:14 +01:00
tek
85abed55b6 Merge branch 'main' of github.com:mvt-project/mvt 2023-05-20 00:14:01 +02:00
tek
2fbd7607ef Adds latest iOS version 2023-05-20 00:11:16 +02:00
Donncha Ó Cearbhaill
3787dc48cd Fix bug where getprop sections where missing due to non-standard section header 2023-05-18 11:28:10 +02:00