Big Data from scratch: Env setup (Part 1)
Recently I have been busy juggling with big data. It is such a deep ocean with wide range of domain knowledge and was particularly difficult for someone who does not have a comsci background. Although there are a lots of resource floating around the internet but not all of them are intuitive. After all the filtering and trial and error I have done through the journey of learning big data ecosystem, thought I should write a comprehensive tutorial not only as a way for my own documentation, but also for people who are in the same boat I was because I never found one single learning resource where it explains things in detailed from the ground up and from scratch! .Hope the stuff I share here would be of use for someone starting the big data journey, as the tech world is moving fast for sure. I remember when I finished my undergraduate degree, we were using SPSS as a “database” and things certainly have changed drastically with new technalogy. Not sure what the future looks like but it is very exciting and scary at the same time.
Some may argue as a statistician or data scientist, one does not need to learn the engineering side of things. Indeed, of course in the real world these tasks will be divided amongst, software engineer, system architect , system admin, ML engineer, data engineer, ETL people, database administrator ,data scientist. However, learning some or even all of them will give you a cohesive view of the entire eco system: e.g. from a single meta data to a fully deployed ML solution. Which might come in handy sooner or later.
Let’s dive in! (Github repository to my work can be found here)
(more…)Dockerize Rshiny app
docker on windows (nightmare)
docker with linux with r (still nightmare but not as much)
steps: pull software, run linux package installation, install shiny, permission configuration, r packages and dependencies
issues encountered: (e.g. Java configuration, different built being run with AMD64）
link:
https://github.com/Z-ingdotnet/Cov_19_dockerized
Building a R shiny app for tracking this 19 Covid epidemic (hope won’t have to change it to pandemic)
Building a shiny app for real time earth quakes visualization
Although R shiny app was around for a long time. I didn’t get the chance to try it out until recently
How to cloud mining cryptocurrencies, and is Cryptocurrency Mining still Profitable？
A confession to be made, I was late on understanding cryptocurrency and by no means am I any expert of the block chain technology. But as I continue to read more about the technology, the more intriguing it gets . I was already developing an interest in cryptography and knowing something that combines cryptography with various other knowledge domains just really excites me. Hence I thought it would be a great opportunity to learn about block chain but learning how its proof of work system works, and thus mining it would be a great place to start.
a side note. I don’t think I am qualified to tell you that cryptocurrency are the best investment you could make. In fact, I’m still very skeptical of it. Although mining cryptocurrencies sounds pretty cool, but don’t be fool by thinking there will be no initial investment. Whether or not cryptocurrency mining is profitable is largely depending on the computing power of your device. And to the best of my knowledge, the current game for popular cryptocurrencies like bitcoins are very large, there are already people making thousands of upfront investment on devices that are specifically designed for cryptocurrency that give them very high hash rates. And mining with CPU just never going to give you the same computational power. However, there are still other cryptocurrencies that are still worthy of a try. At least for educational purposes or just some passive income that you let it run 24/7 along with your website.
I love the “decentralized” concept so lets get started:
Important lessons I learned from traveling/backpacking
中国有句古话：“读万卷书不如行万里路”. It is an old Chinese saying equivalent to “The most valuable lessons in life can never be expressed in black and white, but must be experienced”
I never thought I could just pick up a bag, take my car and disappear. But now looking back, I couldn’t have been more thankful as it was probably one of the best decisions I made in recent years.
life is either a daring adventure or nothing at all–Helen Keller
In university, you get a false perception that everything of importance can be found in books. But actually, many things in life are very hard to put in black and white. There is a reason why many agree that experience is the greatest teacher of all.
I was lucky enough that when I quit my old job, the lease was also about to expire so I took advantage of the situation at the time; hired a storage unit and moved my man cave into it within a day, booked a ferry ticket the same night, took my car and disappear from the city the next day. From then on, for a period of three months I was constantly on the move, from one backpacker to another, one camp to the next, one city to a new one…….This traveling/backpacking experience opened up my eyes and had me lived in a different world that forced me to think and reevaluate what’s important,what to value more and how to live the life I wanted to live. It was a priceless lesson that I think every individual with the opportunity should attend. As Buddha saids: “There are only two mistakes one can make along the road to truth; not going all the way, and not starting.”
Here are the vital lessons traveling taught me:
Text mining classified advertising contents using R, SQL and phpmyadmin (Part I)
wesearch.co.nz was launched back in 2010 as a personal project for learning about the web. I was really proud of the fact that I took something from imagination to execution. However, due to various constrains and other personal commitments at the time, unfortunately the site/idea didn’t receive the maintenance and improvement it deserves. Although the site initially received a fair amount of attention from the public , but there have only been around 300 listings posted to date.
I realized I never had the chance to look at the actual classified ads data stored behind the site. Up until this week, I decided learn some text mining techniques and thought it would be a great idea to extract ads data from the site and use it as sample data.
Disclaimer: This is my first attempt , so by no means I’m a text mining expert, if there are errors or better way to do things please let me know. Below is what I have done:
Why secure Apache server with TLS/SSL certificate and how to do so using Let’s Encrypt–a free, automated, and open Certificate Authority
This weekend I learned something about HTTPS and the reasons for forcing all requests to redirect to https rather than letting requests going through unencrypted http traffic.
In a nutshell, it is about security. Https stands for “Hyper Text Transfer Protocol” with Secure Sockets Layer (SSL). It essentially provides critical security and data integrity for the communications between the website and the users’ browsers. Which in turn protects users’ personal information.
In the absence of a SSL encryption, intruders can exploit unprotected communications through almost every unprotected resource such as images, cookies or scripts to trick users into giving up sensitive information and create security vulnerabilities. They may also passively listen to unprotected HTTP communications between the website and users.
Here is how you set up TLS/SSL certificate on Linux(Ubuntu).
update local apt package indexes and install the Let’s Encrypt client:
sudo apt-get update
sudo apt-get install python-letsencrypt-apache
Leadership is not a journey to rise in the ranks. It is a journey to help those around us rise—Simon Sinek
this is probably a litmus test that most so called leaders fail
Using R for plotting earthquake data and it was shocking
note: plot produced using R, plate boundary and earthquake data(01/Jan/2014-20/Dec/2016) with minimum magnitude of 5 or greater
P.S…never really looked into earthquake data until the recent Kaikorua event happened in NZ and it elevated my curiosity in this space. In a nutshell, although it was obvious that NZ and South America are sitting on several fault lines and that they are in the deforming plate boundary zone. but looking at the number of earthquakes illustrated graphically gave me a bit of a shock.
Have anyone previously applied machine learning in catastrophe/risk management, more specifically, in attempting to predict earthquakes (e.g. using a large number of time series data to put into machine learning algorithms to output a predicted classification)? please let me know
Setting up VPN liked socks5 proxy/ protocols (on a VPS) to reroute network traffic
First time learning these networking stuff ,VPS,socks5 protocols, very interesting learning curve, thoroughly enjoyed it
after generating a public and a private key to ssh, installed the required packages and done some configuration on ports and the firewall etc, I was able to go on Instagram
Factors affecting organizational knowledge creation and sharing in New Zealand firms
Conference Abstract:
PLS17 – Abstract – Final
Below is the Preso I prepared aims to give a quick overview of the entire thesis. Here I am trying to explain the thesis in the form of customer journey mapping, incorporated with some creative and design thinking stuff.
it touches the following domains of knowledge:
- Economics (K-economy theory)
- Knowledge management
- Organisational theory (Dynamic theory of organizational knowledge creation)
- Psychology/Sociology (Social exchange theory, Social processes, Trust,intrinsic and extrinsic rewards&motivation)
- Research Design
- Survey design and pretesting
- Choropleth mapping
- Statistical modelling and analysis (PLS-SEM, Imputation, Q-sort, Content validity checks, Non-responses bias, Item validity tests, Convergent validity, Discriminant validity)
- Web scraping and other Programming
Thanks those who participated the survey for my thesis.
It has been a really tough and long journey to pull this piece of work together which required me to step into uncharted territories to learn and apply new knowledge. Most importantly, the thesis would not have been possible in the absence of the willingness and effort survey participants had put in.
Thankfully the work received high recognition and an A+ grade. My supervisor and I are in the process of turning it into a journal article. Will send all participants relevant link(s) once the publication is complete.
I would like to thank all the participants once again. Prize draw winners have been contacted and all prizes have been sent to the provided address details.
Link to the original survey page is here
