I was tasked to chase down robots for an online auction site. Human bidders on the site are becoming increasingly frustrated with their inability to win auctions vs. their software-controlled counterparts. As a result, usage from the site's core customer base is plummeting.

In order to rebuild customer happiness, the site owners need to eliminate computer-generated bidding from their auctions. The goal of this project is to identify online auction bids that are placed by 'robots', helping the site owners easily flag these users for removal from their site to prevent unfair auction activity.

APPROACH BREAKDOWN

My approach to identify these bids included:

• 1. EDA and Data Imputation: check missing values and fill them with medians.
• 2. Data Manipulation: change data type, feature engineering, generate new features, turn categorical data into dummy/integer variable.
• 3. Apply machine learning models to predict bots.

The machine learning models I used were: Logistic Regression, Random Forest, Gbtree, Xgboost, Lightgbm.

The most useful features I identified were:

• 1. bid frequency over short periods of time.
• 2. average number of bids a user made per auction.
• 3. the entropy for how many bids a user placed on each day of the week.
• 4. maximum number of bids in a 20 min span.
• 5. total number of bids placed by the user.
• 6. minimum and median times between a user's bid and the previous bid by another user in the same auction.
• 7. median delta-time from the last bid until the end of the auction.
• 8. IPs used by a bidder which were also used by another user which was a bot.
• 9. total used IPs, the most used IP, the least used IP, etc.

Nowadays, correction of grammatical errors is a common topic, and there are many tools available for checking spelling/grammatical errors in English, Spanish or Russian. In term of Vietnamese, there is not any tool and research for handling this topic.

Vietnamese is not easy to learn and very complicated, even both Vietnamese locals and Vietnamese learners usually make spelling/grammatical errors in the text. There are several types of error, such as spelling mistakes, using wrong words.

This project is my attempt at building a tool using machine learning to detect and fix spelling/grammatical errors in Vietnamese. This is the most challenging project I have ever done.

APPROACH BREAKDOWN

Approach to build dataset:

• 1. Dataset of error-free sentences: web scraping from multiple open websites, news, blogs in Vietnamese.
• 2. Dataset of error sentences: producing noises (remove, insert, replace, swap letters/words) on error-free sentences to create a dataset of errors.

First Approach:

• 1. Idea: treat the problem as a regression prediction problem. The input is a sentence that is error-free or errors, while the output is the error rate.
• 2. Data Manipulation: calculate the error rates, perform data cleaning for texts, tokenization.
• 3. Train machine learning model on a pretrained PhoBert.
• 4. Predict the error rate of sentences.

Second Approach:

• 1. Idea: treat the problem as a text translation problem. The input is one sentence with errors, while the output is the correct/error-free sentence.
• 2. Data Manipulation: perform data cleaning for texts, tokenization.
• 3. Train machine learning model on a pretrained T5.
• 4. Translate an error sentence to its correct form.

Ask a home buyer to describe their dream house, and they probably won't begin with the height of the basement ceiling or the proximity to an east-west railroad. They might describe how many bedrooms, where it locates, the house age. Based on those criteria, we might want to know the approximate how much that much may cost.

APPROACH BREAKDOWN

Approach to Regression Prediction:

• 1. EDA and Data Imputation: check missing values and fill them with medians.
• 2. Data Visualization: plot feature distribution.
• 3. Data Manipulation: change data types, apply statistic models, create new features, turn categorical data into dummy/integer variables.
• 4. Apply machine learning models to predict house price.

The machine learning models I used were: Lasso, Elastic Net, Support Vector Regression, Gbtree, Xgboost, Lightgbm.

The report is built to answer the following business questions:

• 1. Do we know which features affect the house price most by visualing on graph only and by ML/DL models?
• 2. Can we predict closely to the house price?
• 3. What should we do to have better accuracy in the future, by improving data quality or improving ML/DL algorithms?

Language Adventures is a story-based scavenger-hunt game system that is comprised of a mobile application and a web application.

I worked on this project with 5 other teammates as a Scrum team at the university. I was the team leader, who was responsible for conducting research on the technology, and participated in the whole software development process from design to implementation and delivery. I developed software functionalities on the mobile application to ensure the user journey is not interrupted. Collaborating with peers to perform error analysis and make improvement based on feedback from client and end-users.

PROJECT BREAKDOWN

Mobile Application

• Users of the mobile application are the players, and will be presented with snippets of a story. The players will form teams and need to move around in the real world to reach special story-related locations which will unlock the next part of the story. Players are presented with challenges at specific story-points in the adventure. The Language Adventures mobile app has the capability to house many different stories.

Website Application

• Users of the website application are the game masters. The game masters monitor and assist the players (users of the mobile application). The website application allows the game master to see location of any of the currently playing teams and send messages to them. The website will also control the creation of teams and stories.

The Technologies:

• 1. Mobile application, written in C# and using the Xamarin platform, specifically Xamarin forms to allow cross-platform.
• 2. Website application and the Web API. The Web API facilitates communication between the database and applications. Both Web API and website are written in C#. The Web API uses the ASP.NET framework. The Website application uses the .NET core framework.

The MVP:

This is a post dataset that contains 59 posts occurring between 2018-01-12 and 2020-10-28 for a marketing agency. The company mainly does digital advertising. Many customers of the company are small-to-middle promising brands. We will extract the insights by analyzing many factors that influence the performance of these posts such as Reach, Impression, Like, Comment, Share, etc.

APPROACH BREAKDOWN

Approach to Post Analysis:

• 1. EDA and Data Imputation: check missing values and fill them with medians.
• 2. Data Manipulation: format date, extract new features such as day, month, hour, day of week.
• 3. Data Visualization: plot feature distribution.
• 4. Compare the values and conclude the insights.

The report is built to answer the following business questions:

• 1. Which is the top 5 most Reach / Impression / Engagement posts?
• 2. What is the Impression / Engagement performance over 24h day?
• 3. What is the Impression / Engagement performance over a 7-day week?
• 4. What is the Impression / Engagement performance over a 12-month year?
• 5. What is the relationship between Impression and Engagement?
• 6. How do the literature factors such as Keywords, Phrases, Hashtags affect the Impression / Engagement?
• 7. Which type of content (text, image, video) has better performance?

In this project, I developed a machine learning tool for classifying music audio into different genres. I built 2 different tools from 2 machine learning approaches and then compared their performances.

The dataset consists of 1000 audio tracks each 30 seconds long. It contains 10 music genres, each represented by 100 tracks. The tracks are all 22050Hz Mono 16-bit audio files in .wav format.

APPROACH BREAKDOWN

First Approach:

• 1. Library: Librosa, Keras.
• 2. Extract audio features including RMS, waveform chromagram, spectral centroid, bandwidth, roll-off frequency, zero-crossing rate, MFCCs.
• 3. Store features into dataset for train.
• 4. Apply artificial neural network (ANN) model to train the classifier.

Second Approach:

• 1. Library: Librosa, Keras.
• 2. Extract Mel-scaled spectrogram images from audio.
• 3. Prepare dataset with image data generator.
• 4. Apply convolutional neural network (CNN) model to train the classifier.

This is a project about a movies recommender system, which replicates the industry use case. I developed this recommender system when I practiced the multiple concepts and techniques of recommendation. The system is well organized and allows customization on various supportive algorithms/filterings, which you can easily modify and apply in different use cases or objectives (e.g. movies, books, shopping items, music, video games, etc).

PROJECT BREAKDOWN

Recommender metrics: RMSE, MAE, HR, CHR, ARHR, Coverage, Diversity, Novelty.

Recommender similarity score: Cosine, Multi-dimensional Cosine, Adjusted Cosine, Time Similarity, Mean Squared Difference, Pearson, Jaccard, Spearman Rank Correlation.

Recommender filters:

• 1. Content-based filtering.
• 2. Collaborative neighborhood-based filtering: user-based, item-based.
• 3. Matrix factorization filtering: PCA, SVD.
• 4. Neural network filtering: RBM, AutoEncoder, RNN.

The first campaign aims to acquire and retain users, convert them into frequent users of MoMo, an online wallet platform. Could we evaluate the performance of the campaign? Is there any interesting pattern in the data? And finally can we do anything about the secret we just found?

The second campaign aims to know if the different MoMo ages have different serviceId cluster, and how we do cross-sale to customers.

APPROACH BREAKDOWN

The report is built to answer the following business questions for the first campaign:

• 1. What is the ratio of churn and retain customers?
• 2. Which shops drive the most customers?
• 3. Which location customers are more likely to retain?
• 4. Which shops have the highest retention rate?
• 5. Which shops inherited both characteristics from Super Market and Convenience Store?
• 6. Which combination of features derives the insights from customers' Status?
• 7. What are the unnecessary features that we can remove?
• 8. What are the important features for prediction capability?
• 9. Whether using SHAP will help us realize how variables impact customer' Status?

The report is built to answer the following business questions for the second campaign:

• 1. Whether users of different MoMo ages have different serviceId cluster?
• 2. Which serviceId we can cross-sales to users?

This is a transnational data set that has information of 100k orders from 2016 to 2018 made at multiple marketplaces in Brazil. Its features allow viewing an order from multiple dimensions: from order status, price, payment and freight performance to customer location, product attributes and finally reviews written by customers.

APPROACH BREAKDOWN

Approach to E-Commerce Analytics:

• Data cleaning and Feature engineering: remove outliers, extract new features from InvoiceDate such as date, day, month, year, hour, day of week.

The report is built to answer the following business questions:

• 1. Who, customerID, bring most revenue?
• 2. Who, customerID, buy most in term of quantity?
• 3. Who, customerID, is likely to return the product?
• 4. Which item is bought most and least?
• 5. Which country bring most revenue in total and average?
• 6. Which month we sell out most and least?
• 7. What time people tend to buy our product?
• 8. Which day of a week people tend to visit and purchase stuff?
• 9. Are there any relationship between Repeat Customers and All Customers over a year?
• 10. What is the most trending of some items?
• 11. Top 10 Reordered Items?
• 12. What is the Mall's Cancellation Rate?
• 13. The revenue comes from repeat items or 1 items per month?
• 14. What is the frequent itemsets? By applying Association Rule to understand market basket?
• 15. How many customer types, based on RFM model?
• 16. How to understand customer acquisition from marketing campaign, by conducting Cohort analysis?

A fast food chain plans to add a new item to its menu, however between 3 possible marketing campaigns they undecided the one that might bring the greatest effect on sales. Some basic data we have is MarketID, Market Size, Location, Store Age, Promotion, Week, Sales. We will find figure out which promotion/campaign is the best in this case.

APPROACH BREAKDOWN

Approach to A/B Test:

• 1. EDA and Data Imputation: check missing values and fill them with medians.
• 2. Data Visualization: plot feature distribution.
• 3. Performing A/B Test, by applying ANOVA(F-Test), Z-Test to point out the best campaign among 3.
• 4. Applying machine learning model to extract important features from the dataset.

The report is built to answer the following business questions:

• 1. What is the sale in each MarketID?
• 2. What is the sale in each MarketSize?
• 3. What is the interesting points in sale from each pair of MarketID and MarketSize?
• 4. What is the relationship between MarketSize and Store Age?
• 5. How Promotions derived from MarketSize have effect on Sales?
• 6. The longer we run the campaign, the more revenue it come?

Rossmann operates over 3,000 drug stores in 7 European countries. Store sales are influenced by many factors, including promotions, competition, school and state holidays, seasonality, and locality. Currently, Rossmann store managers are tasked with predicting their daily sales for up to six weeks in advance.

APPROACH BREAKDOWN

Approach to Sale Forecasting:

• 1. EDA and Data Imputation: check missing values and fill them with medians.
• 2. Data Visualization: plot feature distribution and ECDF.
• 3. Data Manipulation: change data types, impute data, create new features.
• 4. Apply ARIMA, Prophet Facebook and other traditional machine learning models to extract insights.

The report is built to answer the following business questions:

• 1. What is the sale trend over time?
• 2. What is the number of customers come to stores across a week / month / year?
• 3. What is the number of sales of stores across a week / month / year?
• 4. Does promotion has positive influences on the number of customers coming to stores and revenue overall?
• 5. What is the sale on holidays vs non-holiday?
• 6. Determine how significant features influence on sale?
• 7. Can we closely predict the sale for the following months?

Are you tired of manually copying and pasting values in a spreadsheet? Do you want to learn how to obtain interesting, real-time and even rare information from the internet with a simple script? When it comes to data science – more and more data comes from external sources, thus knowing how to extract and structure that data quickly is an essential skill that will set you apart in the job market.

APPROACH BREAKDOWN

Approach to Web Scraping:

• 1. Choose Library: BeautifulSoul.
• 2. Find an element containing all data.
• 3. Extract the desired data from the web page.
• 4. Data Manipulation: clean data, fill missing values.
• 5. Store data in structured form: Excel, Database, Powerpoint.

The report is built to answer the following business questions:

• 1. Why we need Web scraping?
• 2. Is it legal or illegal to crawl data from an arbitrary website?
• 3. If we have thousands of urls, how do we speed up the process?

DATA INSIGHTS ANALYSIS

• 1. Email Marketing Analysis
• 2. Marketing Campaign Analysis
• 3. Regional Profit Analysis
• 4. World Demographics - 50 Years
• 5. Clothing Retailing Expansion Analysis
• 6. Coal Terminal Maintenance
• 7. Startup Quadrant
• 8. Laundromat Startup Expansion
• 9. UK Bank Customers

Go is an abstract strategy board game popular in East Asian countries. The game is invented for two players, in which the aim is to surround more territory than the opponent.

This project replicated a mini version of the Go game that I developed for Window using C++. This is the very first IT project I have done since my first day of learning to code. It brought the laughing and memory every time I read back these 'naive' lines of code.

AWS system of an E-Commerce application Cadabra.com

SYSTEM BREAKDOWN

1. Order History:

• - Serverlogs > KDS > Lambda > DynamoDB > Client App
• - Order history function simulate order data being generated from logs on EC2 instance and publish that data using Kinesis Data Streams into an AWS Lambda function which in turn will populate an order database in DynamoDB that the app can read from.

2. Product Recommendations:

• - Serverlogs > KDF > S3 > EMR
• - Automated product recommendations which involves training an ALS machine learning model based on aggregate customer purchasing data. To build this we will publish our order data through Kinesis Data Firehose into a busket S3 and spin up an AWS EMR cluster to produce recommendations model using Apache Spark.

3. Transaction Rate Alarm:

• - Serverlogs > KDS > KDA > KDS > Lambda > SNS
• - An real-time operational system that alerts if an unexpected rate of orders comes in all of a sudden. To build this we will use Kinesis Data Streams and Kinesis Data Analytics to monitor our incoming orders and use a Lambda function to fire off alarms using AWS SNS to mobile phone when something unusual happens.

4. Log Analysis:

• - Serverlogs > KDF > Elasticsearch Service
• - Log Analysis system to analyze server logs data in near real time for operational purposes. For this, we will use Kinesis Data Firehose to pump Apache logs directly into AWS Elasticsearch where we can query that data and build analysis dashboards using Kabana.

5. Data Warehousing & Visualization:

• - Serverlogs > KDF > S3 > (1)/(2)
• - (1-serverless) > Glue > Athena
• - (2-managed) > Redshift > QuickSight
• - Data Warehousing and Visualization that need for business analysis purposes by using AWS Glue, Athena and Redshift, Quicksight on top of S3 data lake.