flow-indexer - Flow-Indexer indexes flows found in chunked log files from bro,nfdump,syslog, or pcap files

  •        19

The deciding factor for how to partition the databases is how many unique ips you see per day. I suggest starting with monthly indexes. If the indexing performance takes a huge hit by the end of the month, switch to daily indexes. the indexall command will expand file_glob and index any log file that matches.

https://github.com/JustinAzoff/flow-indexer

Tags
Implementation
License
Platform

   




Related Projects

tcpreplay - Pcap editing and replay tools for *NIX and Windows - Users please download source from

  •    C

Tcpreplay is a suite of GPLv3 licensed utilities for UNIX (and Win32 under Cygwin) operating systems for editing and replaying network traffic which was previously captured by tools like tcpdump and Ethereal/Wireshark. It allows you to classify traffic as client or server, rewrite Layer 2, 3 and 4 packets and finally replay the traffic back onto the network and through other devices such as switches, routers, firewalls, NIDS and IPS's. Tcpreplay supports both single and dual NIC modes for testing both sniffing and in-line devices.Tcpreplay is used by numerous firewall, IDS, IPS, NetFlow and other networking vendors, enterprises, universities, labs and open source projects. If your organization uses Tcpreplay, please let us know who you are and what you use it for so that I can continue to add features which are useful.

Blacklight - Discovery Interface for any Apache Solr

  •    Ruby

Blacklight is an open source OPAC (online public access catalog). It is ruby-on-rails based discovery interface (a.k.a. “next-generation catalog”) especially optimized for heterogeneous collections. It could be used as a library catalog, as a front end for a digital repository, or as a single-search interface to aggregate digital content that would otherwise be siloed. Blacklight uses Solr, an enterprise-scale index for its search engine.

ElasticSearch - Distributed, RESTful search and analytics engine

  •    Java

Elasticsearch is a distributed, RESTful search and analytics engine capable of solving a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data so you can discover the expected and uncover the unexpected.

joy - A package for capturing and analyzing network flow data and intraflow data, for network research, forensics, and security monitoring

  •    C

Joy is a BSD-licensed libpcap-based software package for extracting data features from live network traffic or packet capture (pcap) files, using a flow-oriented model similar to that of IPFIX or Netflow, and then representing these data features in JSON. It also contains analysis tools that can be applied to these data files. Joy can be used to explore data at scale, especially security and threat-relevant data. JSON is used in order to make the output easily consumable by data analysis tools. While the JSON output files are somewhat verbose, they are reasonably small, and they respond well to compression.

elastiflow - Network flow Monitoring (Netflow, sFlow and IPFIX) with the Elastic Stack

  •    Shell

ElastiFlow™ provides network flow data collection and visualization using the Elastic Stack (Elasticsearch, Logstash and Kibana). It supports Netflow v5/v9, sFlow and IPFIX flow types (1.x versions support only Netflow v5/v9). The following dashboards are provided.


diskover - File system crawler, disk space usage, file search engine and file system analytics powered by Elasticsearch

  •    Python

diskover is an open source file system crawler and disk space usage software that uses Elasticsearch to index and manage data across heterogeneous storage systems. Using diskover, you are able to more effectively search and organize files and system administrators are able to manage storage infrastructure, efficiently provision storage, monitor and report on storage use, and effectively make decisions about new infrastructure purchases. As the amount of file data generated by business' continues to expand, the stress on expensive storage infrastructure, users and system administrators, and IT budgets continues to grow.

NFDUMP - Netflow processing tools

  •    C

nfdump is a set of tools to collect and process netflow data. It's fast and has a powerful filter pcap like syntax. It supports netflow versions v1, v5, v7, v9 and IPFIX as well as a limited set of sflow and is IPv6 compatible. For CISCO ASA devices, which export Netflow Security Event Loging (NSEL) records, please use nfdump-1.5.8-2-NSEL.

Constellio - Enterprise Search engine

  •    Java

Constellio Open Source Enterprise Search is based on Apache Solr and using Google Search Appliances connectors architecture, it allows, with a single click, to find all relevant content in your organization (Web, email, ECM, CRM etc.).

tranalyzer

  •    C

This tool generates extended netflow-like flow statistics from large pcap files or extensive ethernet interface measurements. It is intended to serve as an IT troubleshooting tool and a pre-processing for scientific analysis tools.

Perl File Indexer

  •    Perl

Perl File Indexer is a tool which can index a specific set of files, (given by a list of extensions) which you can later search using incremental search. Once a file is launched, it will be opened using the system default application for the extension.

Sphinix - Search server

  •    C++

Sphinix is free open-source SQL full-text search engine. How do you implement full-text search for that 10+ million row table, keep up with the load, and stay relevant? Sphinx is good at those kinds of riddles.

Yioop - Open Source Search Engine Software

  •    PHP

Yioop is an open source, PHP search engine capable of crawling, index, and providing search results for hundred of millions of pages on relatively low end hardware. It can index a variety of text formats HTML, RSS, PDF, RTF, DOC and images GIF, JPEG, PNG, etc. It can import data from ARC, WARC, Media-Wiki, Open Directory RDF. It is easily localized to many languages. It has built-in support for new feeds, discussion groups, blogs, and wikis. It also supports mixing indexes to create mash ups.

ASPseek

  •    C++

ASPseek is an Internet search engine software developed by SWsoft.ASPseek consists of an indexing robot, a search daemon, and a CGI search frontend. It can index as many as a few million URLs and search for words and phrases, use wildcards, and do a Boolean search. Search results can be limited to time period given, site or Web space (set of sites) and sorted by relevance (PageRank is used) or date.

Indexer

  •    

A desktop file Index and Search tool which allows you to choose a list of folders to index, and then search on later. It is based on Lucene.net and the IFilter mechanism.

Solr - Blazing-fast, open source enterprise search platform

  •    Java

Solr is the popular, blazing fast open source enterprise search platform from the Apache Lucene project. Its major features include powerful full-text search, hit highlighting, faceted search, dynamic clustering, database integration, and rich document (e.g., Word, PDF) handling. Solr is highly scalable, providing distributed search and index replication, and it powers the search and navigation features of many of the world's largest internet sites.

Lux - XML Search engine

  •    Java

Lux is an open source XML search engine using Lucene /Solr and Saxon XQuery/XSLT processor. Lux provides XML-aware indexing, an XQuery 1.0 optimizer that rewrites queries to use the indexes, and a function library for interacting with Lucene via XQuery. These capabilities are tightly integrated with Solr, and leverage its application framework in order to deliver a REST service, application server, and supporting tools.

compass - Searchengine built on top of Lucene

  •    Java

Compass is a real time searchengine. It is built on top of lucene. It is transactional, distributed, supports Spring MVC, integrates with Hibernate.

Peter C# Lucene Indexer

  •    

C# application, can index and search the folder. It will check the file last modified date before index, if the date chanage, it will index the file, otherwise not.

loklak_search - Frontend Search for loklak server http://loklak.org

  •    TypeScript

The loklak_search creates a website using the loklak server as a data source. The goal is to get a search site, that offers timeline search as well as custom media search, account and geolocation search. In order to run the service, you can use the API of https://api.loklak.org or install your own loklak server data storage engine. loklak_server is a server application which collects messages from various social media tweet sources, including Twitter. The server contains a search index and a peer-to-peer index sharing interface. All messages are stored in an elasticsearch index.

Moloch - Large scale, full packet capturing, indexing, and database system

  •    Javascript

Moloch is an open source, large scale, full packet capturing, indexing, and database system. Moloch augments your current security infrastructure to store and index network traffic in standard PCAP format, providing fast, indexed access. An intuitive and simple web interface is provided for PCAP browsing, searching, and exporting.





We have large collection of open source products. Follow the tags from Tag Cloud >>


Open source products are scattered around the web. Please provide information about the open source projects you own / you use. Add Projects.