More precisely, I crawled 250,113,669 pages for just under 580 dollars in 39 hours and 25 minutes, using 20 Amazon EC2 machine instances.

I carried out this project because (among several other reasons) I wanted to understand what resources are required to crawl a small but non-trivial fraction of the web. In this post I describe some details of what I did. Of course, there’s nothing especially new: I wrote a vanilla (distributed) crawler, mostly to teach myself something about crawling and distributed computing. Still, I learned some lessons that may be of interest to a few others, and so in this post I describe what I did. …


Image for post
Image for post

You are given an array of distinct integers arr and an array of integer arrays pieces, where the integers in pieces are distinct. Your goal is to form arr by concatenating the arrays in pieces in any order. However, you are not allowed to reorder the integers in each array pieces[i].

Return true if it is possible to form the array arr from pieces. Otherwise, return false.

Example 1:

Input: arr = [85], pieces = [[85]]
Output: true

Example 2:

Input: arr = [15,88], pieces = [[88],[15]]
Output: true
Explanation: Concatenate [15] then [88]

Example 3:

Input: arr = [49,18,16], pieces = [[16,18,49]]
Output: false
Explanation: Even though the numbers match, we cannot reorder pieces[0]. …


We have n cities labeled from 1 to n. Two different cities with labels x and y are directly connected by a bidirectional road if and only if x and y share a common divisor strictly greater than some threshold. More formally, cities with labels x and y have a road between them if there exists an integer z such that all of the following are true:

  • x % z == 0,
  • y % z == 0, and
  • z > threshold.

Given the two integers, n and threshold, and an array of queries, you must determine for each queries[i] = [ai, bi] if cities ai and bi are connected (i.e. …


The general answer is to 1) maintain a queue of 10,000 jobs, each one is to query a machine’s status 2) have a bunch of worker machines run in parallel, retrieving and executing one status querying job at a time.

Following are the steps of getting deeper in this question and ultimately reach a satisfying state.

Step 1 — Naive Approach

A pitfall when the interviewer presents this question is that, he won’t tell you how many machines you can use to do such an operation. Thus, the first impression an inteviewee got is it can be a dead simple a loop.

List<Status> getStatus(List<Host> hosts) {
List<Status> status = Lists.newArrayList();
hosts.forEach(h -> {
status.add(h.getStatus()); …


Image for post
Image for post

What happens when we start a program and pass parameter?

We can start a program and pass parameter as a command line argument. We can read it those passing parameter easily by rust provided std::env module. Here is an example:


Image for post
Image for post

Inspiration: One of my friend recently called and told me he is facing a problem. He is trying to set up some environment variable in the server. He sshed into the machine and set up an environment variable. He does not see those environment variables after closing the session and connected back again. More precisely, app server does not see those env variable. I am writing these to explain what happens behind the scene and why does it happen.

What happens when you ssh into a machine?

Basically, you get the access to a shell of the machine securely.

What is shell? …


Sockets are a method of IPC that allows data to be exchanged between application in same host or remote host. In the telephonic world, if i want to communicate with my friend, I need a telephone and my friend needs a telephone as well. In the telephonic world, “telephone” is the socket.

In typical client-server application, applications communicate by using sockets.

Image for post
Image for post
  • Each application creates a socket. A socket is the apparatus that allows communication and both applications needs socket.
  • Server binds socket to a well-known address(IP address) and accept incoming connection request
  • Client connects to the server socket.

How to create a socket, in general, we depend on the system call for that. …


For all the people frustrated by having to use to_string() to get programs to compile this post is for you. For those not quite understanding why Rust has two string types String and &str, I hope to shed a little light on the matter.

Functions That Accept A String

I want to discuss how to build interfaces that accept strings. I am an avid hypermedia fan and am obsessed about designing interfaces that are easy to use. Let’s start with a method that accepts a String. Our search hints that std::string::String is a good choice here.

fn print_me(msg: String) {
println!("the …


There are 3n piles of coins of varying size, you and your friends will take piles of coins as follows:

  • In each step, you will choose any 3 piles of coins (not necessarily consecutive).
  • Of your choice, Alice will pick the pile with the maximum number of coins.
  • You will pick the next pile with maximum number of coins.
  • Your friend Bob will pick the last pile.
  • Repeat until there are no more piles of coins.

Given an array of integers piles where piles[i] is the number of coins in the ith pile.

Return the maximum number of coins which you can have. …

About

Omar Faroque

Enterprise Applications Architectecture | JAVA | Rust |Parallel & Distributed Programing | Node.js | IoT | Golang |Starter|Microservices

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store