10 sites to get the large data set or data corpus for free

We have collection of more than 1 Million open source products ranging from Enterprise product to small libraries in all platforms. We aggregate information from all open source repositories. Search and find the best for your needs. Check out projects section.

10 sites to get the large data set or data corpus for free

We aggregate and tag open source projects. We have collections of more than one million projects. Check out the projects section.

You may require GBs of data to do performance or load testing. How your app behaves when there is loads of data. You need to know the capacity of your application. This is the frequently asked question from the sales team "The customer is having 100GB of data and he wants to know whether our product will handle this? If so how much RAM / Disk storage required?". This article has pointers to the large data corpus.

How to generate that data? The easiest way would be to have some samples of data, multiply it using some scripts. Another option would be to create data using random values. The main disadvantage of this approach is the data will have very less unique content and it may not give desired results. Below are some links to get large data set.

Wikipedia:Database, Wikipedia offers free copies of all available content to interested users. data is available in multiple languages. Content along with images could be downloaded. http://en.wikipedia.org/wiki/Wikipedia:Database_download

Common crawl builds and maintains an open crawl of the web accessible to everyone. The data is stored in amazon s3bucket and the requester may have spend some money to access it. https://www.commoncrawl.org/

EDRM File Formats Data Set, consists of 381 files covering 200 file formats. http://www.edrm.net/resources/data-sets/edrm-file-format-data-set

Apache Mahout TLP project to create scalable, machine learning algorithms. Mahout has many links to get free and paid corpus data. https://cwiki.apache.org/confluence/display/MAHOUT/Collections

EDRM Enron Email Data Set v2 consist of Enron e-mail messages and attachments in two sets of downloadable compressed files: XML and PST. http://www.edrm.net/resources/data-sets/edrm-enron-email-data-set-v2

ClueWeb09 dataset was created to support research on information retrieval and related human language technologies. It consists of about 1 billion web pages in ten languages that were collected in January and February 2009. The dataset is used by several tracks of the TREC conference. http://lemurproject.org/clueweb09/

DMOZ - Open Directory Project is the largest, most comprehensive human-edited directory of the Web. It has collections of URLs in different category. Dmoz is one main source for internet search engines. http://www.dmoz.org/rdf.html

theinfo.org - This is a site for large data sets and the people who love them: the scrapers and crawlers who collect them, the academics and geeks who process them, the designers and artists who visualize them. It's a place where they can exchange tips and tricks, develop and share tools together, and begin to integrate their particular projects. http://theinfo.org/

Project Gutenberg offers over 36,000 free ebooks to download to your PC, Kindle, Android, iOS or other portable device. http://www.gutenberg.org/

Million song data set, has data related to tracks and artist. http://labrosa.ee.columbia.edu/millionsong/pages/additional-datasets

Mailing list archive, Subscribe to any mailing list, you will get dozens of emails. Many groups or communities provide option to download the mail archive.

Sometimes we may not get the type or format of data we want. In this situitation, we could get these data, write some script to convert it to our desired format.

This large data set helps to do load test your app and understand its capacity and bottleneck. Using these data set you cannot validate the test results. If you build a search engine, you cannot verify that these many number of hits should be returned for a given keyword.

Now every company is moving towards cloud. People talk about big data but there is some way to generate these data, so that the application could be well tested.

Findbestopensource Editorial Team

We publish blog post about open source products. If you are interested in sharing knowledge about open source products, please visit write for us

Subscribe to our newsletter.

We will send mail once in a week about latest updates on open source tools and technologies. subscribe our newsletter

8 Best Open Source Searchengines built on top of Lucene

Lucene is most powerful and widely used Search engine. Here is the list of 7 search engines which is built on top of Lucene. You could imagine how powerful they are.

Introduction to Apache Cassandra

Apache Cassandra was designed by Facebook and was open-sourced in July 2008. It is regarded as perfect choice when the users demand scalability and high availability without any impact towards performance. Apache Cassandra is highly scalable, high-performance distributed database designed to handle large voluminous amounts of data across many commodity servers with no failure.

React Window - Optimized way to display large list

In Web development, there will be always a need to display content in a large list. Rendering a large list will slow down the web page load and increase the First Contentful Paint(FCP). Usually large lists are displayed in a paginated grid where user has to move around pages by clicking next. It will be good user experience, if the same page provides data of large list as and when we scroll.

Exonum Blockchain Framework by the Bitfury Group

Exonum is an extensible open source blockchain framework for building private blockchains which offers outstanding performance, data security, as well as fault tolerance. The framework does not include any business logic, instead, you can develop and add the services that meet your specific needs. Exonum can be used to build various solutions from a document registry to a DevOps facilitation system.

AbanteCart - Easy to use open source e-commerce platform, helps selling online

AbanteCart is a free, open source shopping cart that was built by developers with a passion for free and accessible software. Founded in 2010 (launched in 2011), the platform is coded in PHP and supports MySQL. AbanteCart’s easy to use admin and basic layout management tool make this open source solution both easy to use and customizable, depending on the skills of the user. AbanteCart is very user-friendly, it is entirely possible for a user with little to no coding experience to set up and use this cart. If the user would be limited to the themes and features available in base AbanteCart, there is a marketplace where third-party extensions or plugins come to the rescue.

Data Fetching and Form Building using NextJS

Next.js is one of the easy-to-learn frameworks for server-side pre-render pages for client-side web applications. In this blog, we will see how we can fetch data from API and make it pre-render pages. Also, let's see how forms work in Next.js and collect the data without maintaining the database.

PySpark: Installation, RDDs, MLib, Broadcast and Accumulator

We knew that Apace Spark- the most famous parallel computing model or processing the massive data set is written in Scala programming language. The Apace foundation offered a tool to support the Python in Spark which was named PySpark. The PySpark allows us to use RDDs in Python programming language through a library called Py4j. This article provides basic introduction about PySpark, RDD, MLib, Broadcase and Accumulator.

How Debuggers Work

The first thing I do when I create a project is to create the debugger launch config at the `.vscode` folder. Debuggers help me to avoid putting print statements and building the program again. I always wondered how a debugger can stop the program on the line number I want and be able to inspect variables. Debugger workings have always been dark magic for me. At last, I managed to learn dark art by reading several articles and groking the source code of delve.

Zingg - Open source deduplication, data mastering and entity resolution

As enterprises struggle to save and analyze more and more data, data quality suffers. Real world data contains multiple records belonging to the same customer. These records can be in single or multiple systems and they have variations across fields which makes it hard to combine them together, especially with growing data volumes. This hurts customer analytics and no AI algorithm for segmentation can produce right results.

An introduction to LucidWorks Enterprise Search

Lucidworks Enterprise search solution is built on top of Apache Solr. It scales seamlessly w/sub-second response times under extreme query loads for multi-billion document collections. It has user friendly UI, which does all the job of configuration and search.

Best open source Text Editors

Text editors are mainly used by programmers and developers for manipulating plain text source code, editing configuration files or preparing documentation and even viewing error logs. Text editors is a piece of software which enables to create, modify and delete files that a programmer is using while creating website or mobile app.In this article, we will discuss about top 7 all-round performing text editors which is highly supportive for programmers.

Getting Started with Spring Batch

The best way to design a system for handling bulk workloads is to make it a batch system. If we are already using Spring, it will be easy to add a Spring Batch to the project. Spring batch provides a lot of boiler plate features required for batch processing like chunk based processing, transaction management and declarative input/output operations. It also provides job control for start, stop, restart, retry and skip processing also.

Enhancing The Experience With Android TV: Briefly About One Developer's Impressions

In my software development career, I have always been attracted to new technologies and innovative solutions. Project by project, I have been exploring something new, discovering and implementing different approaches and trying new solutions. When Android TV showed up, I set a new personal goal. I described my impressions and the overall opinion on the application development for Android TV right here. Hope you will find it useful and interesting.

Holistic usage guide for OpenSSL

OpenSSL is a general purpose cryptographty toolkit that provides an open source implementation of Transport Layer Security(TLS) and Secure Socket Layer(SSL) protocols. It is written in C,assembly and Perl language but wrappers are available in all languages. This article explains about OpenSSL commands.

Data dumping through REST API using Spring Batch

Most of the cloud services provide API to fetch their data. But data will be given as paginated results as returning the complete data will overshoot the response payload. To discover the complete list of books or e-courses or cloud machine details, we need to call the API page-wise till the end. In this scenario, we can use Spring Batch to get the data page by page and dump it into a file.

Connect to MongoDB and Perform CRUD using Java

MongoDB is a popular and widely used open source NoSQL database. MongoDB is a distributed database at its core, so high availability, horizontal scaling, and geographic distribution is quite possible. It is licensed under Server Side Public License. Recently they moved to Server Side Public License, before that MongoDB was released under AGPL. This article will provide basic example to connect and work with MongoDB using Java.

Blockchain Technology Will Revolutionize These Five Industries

Blockchain is buzzing across the world with its endless potential for disrupting a broad spectrum of the industries beyond the storage and transferring the values. Cryptocurrencies are revolutionizing variant industries and plenty of people talk about it with and their potential. Blockchain technology may have been made for bitcoins but it has immense potential for making the transactions secure and exciting. The technology is not ubiquitous but it already seems to be disrupting a huge number of the highly centralized industries that demonstrate practical use cases for changing the way businesses are done.

Solr vs Elastic Search

Solr and Elastic Search are built on top of Lucene. Both are open source and both have extra features which makes programmer life easy. This article explains the difference and the best situation to choose between them.

Laravel Paypal Integration - Smart button with server-side integration

You would have seen a lot of blogs for paypal php integration with REST api which is driven completely in the backend. For checkout, paypal provides an easy way to checkout for client side ready-to-use smart button payment. This approach will work only from the frontend, which will not be safe and difficult to reconcile as the backend does not have any information about it. Server side integration with the paypal smart button will help us to reconcile or track the payments even after some issues in the users payment journey. In this blog, we have walkthrough the paypal smart button with server side php laravel integration.

Apache OpenNLP - Document Classification

Apache OpenNLP is a library for natural language processing using machine learning. In this article, we will explore document/text classification by training with sample data and then execute to get its results. We will use plain training model as one example and then training using Navie Bayes Algorithm.