python-haystack - Process heap analysis framework - Windows/Linux - record type inference and forensics

  •        4

Quick usage guide in the docs/ folder. Haystack-reverse CLI in the docs/ folder.

http://trolldbois.blogspot.com/search?q=python-haystack
https://github.com/trolldbois/python-haystack

Tags
Implementation
License
Platform

   




Related Projects

Limon - Limon is a sandbox developed as a research project written in python, which automatically collects, analyzes, and reports on the run time indicators of Linux malware

  •    Python

Limon is a sandbox developed as a research project written in python, which automatically collects, analyzes, and reports on the run time indicators of Linux malware. It allows one to inspect the Linux malware before execution, during execution, and after execution (post-mortem analysis) by performing static, dynamic and memory analysis using open source tools. Limon analyzes the malware in a controlled environment, monitors its activities and its child processes to determine the nature and purpose of the malware. It determines the malware's process activity, interaction with the file system, network, it also performs memory analysis and stores the analyzed artifacts for later analysis.

pyrebox - Python scriptable Reverse Engineering Sandbox, a Virtual Machine instrumentation and inspection framework based on QEMU

  •    C

PyREBox is a Python scriptable Reverse Engineering sandbox. It is based on QEMU, and its goal is to aid reverse engineering by providing dynamic analysis and debugging capabilities from a different perspective. PyREBox allows to inspect a running QEMU VM, modify its memory or registers, and to instrument its execution, by creating simple scripts in python to automate any kind of analysis. QEMU (when working as a whole-system-emulator) emulates a complete system (CPU, memory, devices...). By using VMI techniques, it does not require to perform any modification into the guest operating system, as it transparently retrieves information from its memory at run-time. Several academic projects such as DECAF, PANDA, S2E, or AVATAR, have previously leveraged QEMU based instrumentation to overcome reverse engineering tasks. These projects allow to write plugins in C/C++, and implement several advanced features such as dynamic taint analysis, symbolic execution, or even record and replay of execution traces. With PyREBox, we aim to apply this technology focusing on keeping the design simple, and on the usability of the system for threat analysts.

stackimpact-python - StackImpact Python Profiler - Production-Grade Performance Profiler: CPU, memory allocations, blocking calls, exceptions, metrics, and more

  •    Python

StackImpact is a production-grade performance profiler built for both production and development environments. It gives developers continuous and historical code-level view of application performance that is essential for locating CPU, memory allocation and I/O hot spots as well as latency bottlenecks. Included runtime metrics and error monitoring complement profiles for extensive performance analysis. Learn more at stackimpact.com. Learn more on the features page (with screenshots).

HeapInspector-for-iOS - Find memory issues & leaks in your iOS app without instruments

  •    Objective-C

HeapInspector is a debug tool that monitors the memory heap with backtrace recording in your iOS app. You can discover memory leaks, no longer used objects, abandoned memory and more issues directly on your device without ever starting Instruments. Since ARC has been introduced we don't need to manage the retain & release anymore. ARC is very powerful and makes Objective-C more stable. ARC decreased the number of crashes and improves the memory footprint. ARC is technically doing a powerful job. It knows when to retain, autorelease and release. But ARC doesn't think about the overall architecture how to design for low memory usage. You should be aware that you can still do a lot of things wrong with your memory (even with ARC). You can still get memory pressures or peaks with ARC.


scalloc - A Fast, Multicore-Scalable, Low-Fragmentation Memory Allocator

  •    C++

scalloc provides general-purpose memory allocation involving many threads on many cores can be done with high performance, multicore scalability, and low memory consumption. The main ideas behind the design of scalloc are: uniform treatment of small and big objects through so-called virtual spans, efficiently and effectively reclaiming free memory through fast and scalable global data structures.

Memorylogic - Memory usage logger

  •    Ruby

Adds in proccess id and memory usage in your rails logs, great for tracking down memory leaks.

memorylogic - Adds in proccess id and memory usage in your rails logs, great for tracking down memory leaks

  •    Ruby

The reason I created this was to help track down a memory leak. It worked very well because it adds the process id and memory usage in with EVERY single line logged and at the end of each rails request. By every line, I mean every line in your log will end with “ (mem #{memory_usage})”. This way, if you have a cluster, you can monitor each process and scroll through your logs and watch the memory. If it suddenly starts to jump you can look at the request and at least you have a starting point. You can checkout the code for that specific controller action and try to narrow it down.

xarray - N-D labeled arrays and datasets in Python

  •    Python

xarray (formerly xray) is an open source project and Python package that aims to bring the labeled data power of pandas to the physical sciences, by providing N-dimensional variants of the core pandas data structures. Our goal is to provide a pandas-like and pandas-compatible toolkit for analytics on multi-dimensional arrays, rather than the tabular data for which pandas excels. Our approach adopts the Common Data Model for self- describing scientific data in widespread use in the Earth sciences: xarray.Dataset is an in-memory representation of a netCDF file.

sirius - A distributed system library for managing application reference data

  •    Scala

Sirius is a library for distributing and coordinating data updates amongst a cluster of nodes. It handles building an absolute ordering for updates that arrive in the cluster, ensuring that cluster nodes eventually receive all updates, and persisting the updates on each node. These updates are generally used to build in-memory data structures on each node, allowing applications using Sirius to have direct access to native data structures representing up-to-date data. Sirius does not, however, build these data structures itself -- instead, the client application supplies a callback handler, which allows developers using Sirius to build whatever structures are most appropriate for their application. Said another way: Sirius enables a cluster of nodes to keep developer-controlled in-memory data structures eventually consistent, allowing I/O-free access to shared information.

Hazelcast Jet - Distributed data processing engine, built on top of Hazelcast

  •    Java

Hazelcast Jet is a distributed computing platform built for high-performance stream processing and fast batch processing. It embeds Hazelcast In Memory Data Grid (IMDG) to provide a lightweight package of a processor and a scalable in-memory storage. It supports distributed java.util.stream API support for Hazelcast data structures such as IMap and IList, Distributed implementations of java.util.{Queue, Set, List, Map} data structures highly optimized to be used for the processing

t81_558_deep_learning - Washington University (in St

  •    Jupyter

Deep learning is a group of exciting new technologies for neural networks. Through a combination of advanced training techniques and neural network architectural components, it is now possible to create neural networks of much greater complexity. Deep learning allows a neural network to learn hierarchies of information in a way that is like the function of the human brain. This course will introduce the student to computer vision with Convolution Neural Networks (CNN), time series analysis with Long Short-Term Memory (LSTM), classic neural network structures and application to computer security. High Performance Computing (HPC) aspects will demonstrate how deep learning can be leveraged both on graphical processing units (GPUs), as well as grids. Focus is primarily upon the application of deep learning to problems, with some introduction mathematical foundations. Students will use the Python programming language to implement deep learning using Google TensorFlow and Keras. It is not necessary to know Python prior to this course; however, familiarity of at least one programming language is assumed. This course will be delivered in a hybrid format that includes both classroom and online instruction. This syllabus presents the expected class schedule, due dates, and reading assignments. Download current syllabus.

FASTER - Fast key-value store from Microsoft Research

  •    CSharp

Managing large application state easily and with high performance is one of the hardest problems in the cloud today. We present FASTER, a new concurrent key-value store designed for point lookups and heavy updates. FASTER supports data larger than memory, by leveraging fast external storage. What differentiates FASTER are its cache-optimized index that achieves very high performance — up to 160 million operations per second when data fits in memory; its unique “hybrid record log” design that combines a traditional persistent log with in-place updates, to shape the memory working set and retain performance; and its architecture as an component that can be embedded in cloud apps. FASTER achieves higher throughput than current systems, by more than two orders of magnitude, and scales better than current pure in-memory data structures, for in-memory working sets. FASTER also offers a new consistent recovery scheme that achieves better performance at the expense of slightly higher commit latency. Go to our website for more details and papers.

timeout - A script to measure and limit CPU time and memory consumption of black-box processes in Linux

  •    Perl

The timeout script is a resource monitoring program for limiting time and memory consumption of black-boxed processes under Linux. It runs a command you specify in the command line and watches for its memory and time consumption, interrupting the process if it goes out of the limits, and notifying the user with the preset message. The killer feature of this script (and, actually, the reason why it appeared) is that it not only watches the process spawned directly, but also keeps track of its subsequently forked children. You may choose if the scope of the watched processes is constrained by process group or by the process tree.

rvmi - rVMI - A New Paradigm For Full System Analysis

  •    C

rVMI is a debugger on steroids. It leverages Virtual Machine Introspection (VMI) and memory forensics to provide full system analysis. This means that an analyst can inspect userspace processes, kernel drivers, and pre-boot environments in a single tool. It was specifically designed for interactive dynamic malware analysis. rVMI isolates itself from the malware by placing its interactive debugging environment out of the virtual machine (VM) onto the hypervisor-level. Through the use of VMI the analyst still has full control of the VM, which allows her to pause the VM at any point in time and to use typical debugging features such as breakpoints and watchpoints. In addition, rVMI provides access to the entire Rekall feature set, which enables an analyst to inspect the kernel and its data structures with ease.

pe-sieve - Scans a given process

  •    C++

PE-sieve is a light-weight tool that helps to detect malware running on the system, as well as to collect the potentially malicious material for further analysis. Recognizes and dumps variety of implants within the scanned process: replaced/injected PEs, shellcodes, hooks, and other in-memory patches. Detects inline hooks, Process Hollowing, Process Doppelgänging, Reflective DLL Injection, etc.

Memory Structures Library (MemSL)

  •    C

Easy To Use Collection Classes For C And C++ The Memory Structures Library (MemSL) has implementations of structures (i.e. AVL trees, threaded trees, dynamic hashing, ...) that are extremely hard to find and far harder to implement.

mlens - ML-Ensemble – high performance ensemble learning

  •    Python

ML-Ensemble combines a Scikit-learn high-level API with a low-level computational graph framework to build memory efficient, maximally parallelized ensemble networks in as few lines of codes as possible. ML-Ensemble is thread safe as long as base learners are and can fall back on memory mapped multiprocessing for memory-neutral process-based concurrency. For tutorials and full documentation, visit the project website.

annoy - Approximate Nearest Neighbors in C++/Python optimized for memory usage and loading/saving to disk

  •    C++

Annoy (Approximate Nearest Neighbors Oh Yeah) is a C++ library with Python bindings to search for points in space that are close to a given query point. It also creates large read-only file-based data structures that are mmapped into memory so that many processes may share the same data.To install, simply do sudo pip install annoy to pull down the latest version from PyPI.