crt0mega
/
mvt
mirror of https://github.com/mvt-project/mvt.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity
943 Commits 22 Branches 67 Tags 6 MiB
Commit Graph

25 Commits

Author SHA1 Message Date
tek 6f56939dd7 Requires latest cryptography version 2024-04-15 22:41:01 +02:00
Donncha Ó Cearbhaill 41db117168 Improve performance when checking URLs and domains 
Some MVT modules such as the WhatsApp module can be very slow as it was taking a naive approach to look for IOCs. The code was checking URLs (potentially more than 100k) against
1000's of IOC domains resulting in a quadratic run-time with hundreds of millions of comparisons as the number of IOCs increases.

This commit add an Aho-Corasick library which allows the efficient search in a string (the URL in this case) for all matches in set of keys (the IOCs). This data structure is perfect for this use case.

A quick measurement shows a 80% performance improvement for a WhatsApp database with 100k entries. The slow path is now the time spent fetching and expanding short URLs found in the database. This
can also be sped up significantly by fetching each URL asynchronously. This would require reworking modules to split the URL expansion from the IOC check so I will implement in a separate PR.
 2023-06-29 14:14:44 +02:00
tek a1481683e3 Adapts linter workflow to black 2023-06-14 01:05:14 +02:00
Donncha Ó Cearbhaill 3142d86edd Fix path to include version JSON files in built package 2023-05-21 16:37:36 +01:00
Donncha Ó Cearbhaill 22fd794fb8 Fix python style and setup.cfg syntax 2023-05-21 16:15:49 +01:00
Donncha Ó Cearbhaill bbfaadd297 Load iOS device and build information from a JSON file. 2023-05-20 21:24:14 +01:00
tek 55f6a4ae54 Fixes mypy typing issues 2023-02-21 15:18:36 -05:00
Nex e559fb223b Upgraded dependencies 2022-10-15 11:26:40 +02:00
Nex 2781f33fb5 Added more date conversion wrappers 2022-08-13 14:04:10 +02:00
Nex a935347aed Trying to enforce line lengths at 80/100 2022-08-12 19:14:05 +02:00
Nex 63ff5fd334 Started linting the code 2022-08-12 16:20:16 +02:00
Nex a9eabc5d9d Updated dependencies 2022-07-07 12:28:42 +02:00
Nex 3cdc6da428 Temporarily removed mvt-ios check-usb command 2022-07-06 13:01:55 +02:00
Nex 2fe8b58c09 Removed space 2022-06-30 10:26:30 +02:00
Nex 4a568835d2
Merge branch 'main' into feature/ios-check-usb 2022-06-28 23:58:38 +02:00
tek f864adf97e First structure for mvt-ios check-usb 2022-06-28 20:35:52 +02:00
Nex 815cdc0a88 Adding system to check for updates of indicators files and notify if any are available 2022-06-27 14:41:40 +02:00
Nex a12c4e6b93 First commit to refactor of command definitions 2022-06-15 17:41:19 +02:00
Nex 732a712e3d Changed path where to seek packages 2022-05-08 17:06:34 +02:00
Nex 6d278d4bec Actually, can also move version to setup.cfg 2022-05-08 15:17:19 +02:00
Nex c39b4d2179 Minimized setup.py and moved all possible to setup.cfg 2022-05-08 15:15:00 +02:00
Nex a653fd5253 Moved dependencies to setup.cfg 2022-05-08 15:07:17 +02:00
Nex f754bf274d Unfortunately until #233 is resolved, we have to enforce Python >= 3.8 as well 2022-05-08 14:59:34 +02:00
Nex c343eed5a0 Moved flake8 config to setup.cfg 2022-05-08 14:50:31 +02:00
Nex f61729deed Starting to move declarative information to setup.cfg 2022-05-08 14:45:14 +02:00