I have had this idea in my mind for a while: a social profile aggregator. While everyone uses social media, the consumption of it is normally decentralized. One of the disadvantages is that you have to go to each social network/app and see the updates. In my mind this requires quite a lot of time and we tend to not go to each of the social networks every time (normally because we don’t have time or we forgot).
Other times, you are curious about a movie, tv show or tv serie character. Looking for information on social networks may take you some googling, twitter, instagram and twitter search to know more about it. What about if there is a service that shows you the information in real time? I figured out that what i wanted is a social media profile aggregator where in matter of a second or less I should be able to get an ‘instant’ social media picture of what is going on related to anything I wanted to track.
The MVP is a web application that has some profiles generated through web crawling and manual annotations. Showing information from different social networks and showcasing them all together would be ideal. The aggregation is pre calculated or generated on demand depending of the availability of the profile.
Some examples are:
- Social media about Dexter (tv show):
- My latest feed about memes and fun things to watch:
The tech stack
- Backbone.js for the client side development. AMD modules and code minification with Require.js
- Express for the server side
- Npm for the package manager
- Jenkins for CI
Options for the MVP
During the initial development, one of the things I had in mind for the MVP was to keep it simple (Everyone does that right?).
When you are aggregating social media coming from different sources and keeping it fairly quick you have two options:
- Create a set of crawlers that pull the information for you. The client pull data from the server and renders the data.
- Multiple calls to different apis/providers on the server side. That data is exposed to the client
- Almost realtime. Scheduled jobs on the backend to pull data from the social apis
- Benefit: server has the data for subsequent calls and operations over the data. Good for working and exposing analytics, integrations, etc…
- Disadvantage: complex architecture, near realtime, rate limiting on apis
- Let the client side request calls on demand.
- Multiple calls to different apis/providers on the client side. The client side is in charge of arranging the elements.
- Real time information: basically whatever is exposed on the social apis at a certain time
- Benefits: simpler architecture. Just the js client doing the calls and rendering the information. Also the rate limiting is on the client side, which lets escalate the pull of information as every client has normally a different IP.
- Disadvantages: data is always on the go. Difficult to serve as a sustainable business.
I went for option 2 as it kept the things simple. Option 1 is more for adding value to a business where you provide additional features on top of that such us publishing, engagement, tracking of events, analytics… In my mind, if option 1 goes well and has good feedback, option 1 is just an extension.
It has just been a fun project to code. If I have some time, I would like to create an extensive database about social profiles so anything a person could be searched with decent precision. This could be a business by itself, where you provide social data related to a profile through a REST api.
As a second step towards this project, my idea is to do small iterations of 1 or 2 days where I tackle the following upgrades:
- Crawlers implementation for profile population
- REST api for exposing profiles
- Users management so they can create their own profiles
- Once REST api is exposed, mobile app. The webapp is responsive, but i want this to be as much fluid as possible (no heavy js rendering).
This may take some time as I am focused on other projects but it was good to refresh some concepts. If you have any suggestions, please let me know!