Installing oracle-java9-installer on Ubuntu Error Fix

The oracle-java9-installer, as of writing, has an old URL that doesn’t redirect properly to the right URL, which causes the installer to fail when it tries to download the binaries.

Setting up oracle-java9-installer (9b162-1~webupd8~0) ...
Using wget settings from /var/cache/oracle-jdk9-installer/wgetrc
Downloading Oracle Java 9...
--2017-05-19 04:10:54-- http://www.java.net/download/java/jdk9/archive/162/binaries/jdk-9-ea+162_linux-x64_bin.tar.gz
Resolving www.java.net (www.java.net)... 137.254.56.25
Connecting to www.java.net (www.java.net)|137.254.56.25|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://home.java.net/download/java/jdk9/archive/162/binaries/jdk-9-ea+162_linux-x64_bin.tar.gz [following]
--2017-05-19 04:10:54-- https://home.java.net/download/java/jdk9/archive/162/binaries/jdk-9-ea+162_linux-x64_bin.tar.gz
Resolving home.java.net (home.java.net)... 156.151.59.19
Connecting to home.java.net (home.java.net)|156.151.59.19|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: http://www.oracle.com/splash/java.net/maintenance/index.html [following]
--2017-05-19 04:10:54-- http://www.oracle.com/splash/java.net/maintenance/index.html
Resolving www.oracle.com (www.oracle.com)... 184.30.70.138, 2600:1408:10:184::2d3e, 2600:1408:10:185::2d3e
Connecting to www.oracle.com (www.oracle.com)|184.30.70.138|:80... connected.
HTTP request sent, awaiting response... 503 Service Unavailable
2017-05-19 04:10:54 ERROR 503: Service Unavailable.
download failed
Oracle JDK 9 is NOT installed.

You will need to manually install the binary and run dpkg to configure it. Change the URL from http://www.java.net/download/ to http://download.java.net/.

cd /var/cache/oracle-jdk9-installer
sudo wget http://download.java.net/java/jdk9/archive/162/binaries/jdk-9-ea+162_linux-x64_bin.tar.gz
sudo dpkg --configure -a
Posted in Uncategorized | 2 Comments

Resizing Virtualbox Fixed-Size VDI Disks

As of writing (2017-04-06), there is no native tooling to resize a fixed-size VDI. Those on the internet saying they can resize one are misinformed. You will get this error message:

Progress state: VBOX_E_NOT_SUPPORTED
VBoxManage.exe: error: Resize medium operation for this format is not implemented yet!

To “resize” a fixed-size VDI, it must be cloned to a larger sized VDI.

Step 1 – Create the larger VDI and move data:

Method 1 – Using VBoxManage:

VBoxManage clonehd [old-VDI] [new-VDI] --variant Standard
VBoxManage modifyhd [VDI] --resize [megabytes]
VBoxManage clonehd [new-VDI] [newnew-VDI] --variant Fixed

Disadvantage of this method is you need to make two additional full copies of the disk.

Method 2 – Using Clonezilla:

  1. Create and attach a new, larger fixed-size VDI using the VirtualBox interface.
  2. Attach and boot with a Clonezilla ISO.
  3. Use Device-to-device setting to clone the drive. Remember to press F12 on boot to select CD-ROM.

Step 2 – Expand the underlying partitions:

  1. De-attach Clonezilla ISO and attach GParted ISO.
  2. Boot similarly and resize/move partitions as needed.
Posted in Uncategorized | Leave a comment

Thoughts on Building Serverless Web Applications with Amazon Lambda

I spent several weekends working a project using the Amazon Lambda serverless micro-architecture to see whether it was worth using for larger projects. I created a micro SaaS – https://pdfbatchfill.com – that essentially takes a bunch of rows and spit them out to fields within PDF forms. Here are my brief thought on Lambda:

Complex setup

My overall experience with Lambda was positive; however, I used ClaudiaJS which abstracts nearly all the underlying plumbing. When I first started I did try to set everything up myself through the web interface and I found it overwhelming with the amount of options available due to the fact that Lambda by itself is a generic application “container”. A lot of glue is required for API Gateway and Lambda to expose the endpoints. There’s a lot of little things to do just to have a working route, so I settled on the ClaudiaJS framework to deal with them. There are others like serverless.

Comes fully-loaded

Everything just works when the application is deployed. The endpoints are automatically connected to logging within CloudWatch line by line separated by instances. Having zero need for maintenance on the infrastructure is surprisingly liberating. I am able to focus mostly on the application itself.

Using S3 to host the static page and Cloudflare to handle the DNS, I got an SSL-enabled site for free, assuming a low-traffic site of course.

Limitations due to maturity

AWS services tend to start out very stripped down and Lambda is no exception. One issue, which I encountered, was Lambda’s inability to accept binary form data. Searching the Lambda forum shows that it wasn’t quite ready for general adoption. As in, your project could be SOL and stuck if support wasn’t available for a particular function you need that is only discovered mid-way through. Luckily, I was able to workaround the issue by directly sending binary data through S3.

Posted in Uncategorized | Leave a comment

Ubuntu Linux Email Notification on Hard Disk S.M.A.R.T Errors

This is a short guide on setting up sendmail command on Ubuntu to work with smartmontools to monitor SMART statuses of drives and send email notifications on any failures.

1. Setup sendmail to relay to an external SMTP server (Gmail, hotmail, your own host).

sudo apt-get install postfix

/etc/postfix/main.cf

...
myhostname = yourhostname
relayhost = [yourhosturl]:587
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous
smtp_use_tls = yes
...

/etc/postfix/sasl_passwd

[yourhosturl]:587 username:password
sudo chmod 400 /etc/postfix/sasl_passwd
sudo postmap /etc/postfix/sasl_passwd
sudo service postfix restart

2. Test sendmail.

echo -e "Subject: it works\nYAY!" | sendmail [email protected] \
-F yourhostname

3. Setup smartmontools to monitor drives and send notifications on failure.

sudo apt-get install smartmontools

/etc/smartd.conf

/dev/sda -H -l error -l selftest -f -s (S/../../1/01) -m \
[email protected] -M exec /usr/share/smartmontools/smartd-runner

/etc/default/smartmontools

start_smartd=yes
sudo service smartmontools restart

ref:
https://linux.die.net/man/5/smartd.conf
https://easyengine.io/tutorials/linux/ubuntu-postfix-gmail-smtp/

Posted in Uncategorized | Tagged | Leave a comment

Determining SSD Approximate Remaining Lifespan

I couldn’t find anything readily available that was trustworthy or free to determine the lifespan of an SSD. MTBF isn’t very useful as on-time doesn’t wear down like hard disks do. The next best thing is to get the total amount of bytes written to the drive and compare it with benchmark values, from torture tests (write until dead) in particular, that others have done and published.

The total bytes written is recorded within the SMART database of the drive. SmartMonTools is needed to read it. It is cross-platform and free.

https://www.smartmontools.org/wiki/Download

Once installed, start CMD as an administrator (assuming Windows).

C:\WINDOWS\system32>smartctl -a /dev/sda
smartctl 6.5 2016-05-07 r4318 [x86_64-w64-mingw32-win10] (sf-6.5-1)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Device Model: Crucial_CT525MX300SSD4
Serial Number: 16441483025B
LU WWN Device Id: 5 00a075 11483025b
Firmware Version: M0CR031
User Capacity: 525,112,713,216 bytes [525 GB]
Sector Size: 512 bytes logical/physical
Rotation Rate: Solid State Device
Form Factor: < 1.8 inches
Device is: Not in smartctl database [for details use: -P showall]
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Tue Nov 29 23:36:05 2016 PST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
...
...
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
 1 Raw_Read_Error_Rate 0x002f 100 100 000 Pre-fail Always - 0
 5 Reallocated_Sector_Ct 0x0032 100 100 010 Old_age Always - 0
 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 3
 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 8
171 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0
172 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0
173 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 1
174 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 4
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
184 End-to-End_Error 0x0032 100 100 000 Old_age Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
194 Temperature_Celsius 0x0022 066 042 000 Old_age Always - 34 (Min/Max 28/58)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 100 100 000 Old_age Always - 0
202 Unknown_SSD_Attribute 0x0030 100 100 001 Old_age Offline - 0
206 Unknown_SSD_Attribute 0x000e 100 100 000 Old_age Always - 0
246 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 266902214
247 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 8350815
248 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 352002
180 Unused_Rsvd_Blk_Cnt_Tot 0x0033 000 000 000 Pre-fail Always - 1932
210 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0

We are concerned with three numbers:

  • reallocated sector count
  • sector size – 512
  • total LBA blocks written – 266902214 (in this case, it wasn’t labelled as such; in general, it’s the largest number)

If there are any reallocated sectors, that is bad news as it generally means the drive is on its last legs. Otherwise it can be considered a healthy drive. Multiplying the two numbers yields the total data written in bytes: 512 x 266902214 = 136653933568 bytes or about 127 GB

Referencing various sources on endurance testing, a TLC 240GB drive starts to degrade after 100-1000TB of writes depending on the make. There’s a wide range depending on the generation of technologies (SLC, MLC, TLC) and controllers used.

With the numbers from my SSD, pessimistically, its life is at 127GB / 100TB or 0.1%.

Posted in Uncategorized | Leave a comment

Sony Vaio Pro 13 Disassembled

There aren’t any high resolution images of the Sony Vaio Pro 13 internals, so I thought I would post them since the fan started making rattling noises. One of the reasons I discovered the fan developing the noise is that the fan actually draws air from the air gaps in the keyboard – meaning it is game over if any liquids are spilled on it.

The repair itself was rather simple – disassemble the fan unit, vacuum all the dust, remove the rotor, lubricate the sleeve bearing, replace the thermal compound, and restore everything. The rotor can be forced off by carefully prying it from the back side through the intake openings.

Update:
The data sheet on the ram modules seems to have disappeared since Elpida merged with Micron. I emailed Micron and they sent back the PDF. Elpida J8416E6MB-GN-F (Courtesy of Micron)

img_20161028_1238399 img_20161028_1250588 img_20161028_1250534 img_20161028_1239003 img_20161028_1238486

Posted in Uncategorized | Leave a comment

Setting Up CKAN 2.5 on Ubuntu 14.04

Install root certificates for work https (optional, required if https is intercepted)

sudo mkdir /usr/share/ca-certificates/extra
sudo cp root.crt /usr/share/ca-certificates/extra/root.crt
sudo dpkg-reconfigure ca-certificates

Install CKAN Package

sudo apt-get install -y nginx apache2 libapache2-mod-wsgi libpq5
wget http://packaging.ckan.org/python-ckan_2.5-trusty_amd64.deb
sudo dpkg -i wget python-ckan_2.5-trusty_amd64.deb
rm python-ckan_2.5-trusty_amd64.deb

Install PostgreSQL

sudo apt-get install -y postgresql solr-jetty

/etc/ckan/default/production.ini

solr_url = http://127.0.0.1:8983/solr
ckan.site_url = http://opendata.test

/etc/default/jetty

NO_START=0            # (line 4)
JETTY_PORT=8983       # (line 19)

Add DB User and Init DB

sudo -u postgres createuser -S -D -R -P ckan_default
sudo -u postgres createdb -O ckan_default ckan_default -E utf-8
sudo ckan db init

You should see the CKAN landing page when going to http://localhost

ckan

Create an admin user.

Create a user through the web interface, then use the CLI to promote the user.

sudo ckan sysadmin add yournewusername
Posted in Uncategorized | Leave a comment

Cecilio CEVN-1NA Electric Silent Violin Internal Electronics

I got this relatively cheap electric violin by Cecilio (model CEVN-1NA) and wanted to see what was under the hood. Performance-wise, using a Sony MD 7505 headset, the sound was quite noisy and rather muddled. Tone adjustment just maded the sound even more so and appeared to crackle when turned to the max. Perhaps I may have gotten a defective unit, but the volume was rather minute. I tried several headphones including the stock ones without improvement. Putting the volume adjustment knob to max still yielded under acceptable levels.

Looking at the PCB, there are two main ICs in the built-in amp – one for tone, one for volume.

TDA2822M
MC4558CN

IMG_20160419_2136275

IMG_20160419_2133577

IMG_20160419_2134479

Here’s a rough schematic of how all the components are connected.

img_20161120_2318377

Posted in Uncategorized | Leave a comment

Strongswan PSK IPsec IKEv2 VPN on Ubuntu 14.04 with Blackberry

This is a short guide on configuring Strongswan on Ubuntu 14.04 with pre-shared keys (PSK) for a Blackberry 10 device to connect with. Setting this up requires in-depth knowledge of networking and routing.

Install and configuring Strongswan:

sudo apt-get install strongswan

/etc/ipsec.conf

conn %default
        ikelifetime=60m
        keylife=20m
        rekeymargin=3m
        keyingtries=1
        keyexchange=ikev2
        authby=psk
config setup
        cachecrls=yes
        uniqueids=yes
conn default
    left=%any
    leftsubnet=10.7.100.0/24,192.168.0.0/24,0.0.0.0/0
    leftauth=psk
    right=%any
    rightauth=psk
    rightsendcert=never
    rightsourceip=10.7.100.10/30
    rightdns=8.8.8.8
    auto=add

/etc/ipsec.secrets

%any %any : PSK "yourGatewayPassword"
%any %any : PSK "yourUserPassword"

My server has one network interface connected to LAN and one interface connected directly to the internet. Both have internet routable gateways with the LAN gateway being the default. However, I will use the second interface for VPN connections so a second routing table is required for “same source routing” – allow internet traffic to respond through itself instead of routing through the LAN gateway by default. This may or may not be needed depending on the setup. The easier solution is to use the other gateway as default.

Configure same source routing:

sudo echo 2 ext >> /etc/iproute2/rt_tables

/etc/network/interfaces

# eth0/br0 config
# ...
auto eth1
iface eth1 inet dhcp
   # automatically set up routing table and rules on up and down
   post-up ip route add default via `grep routers /var/lib/dhcp/dhclient.eth1.leases | tail -1 | awk '/routers/ {print $3}' | sed 's/;//'` dev eth1 table ext
   post-up ip rule add from `grep fixed-address /var/lib/dhcp/dhclient.eth1.leases | tail -1 | awk '/fixed-address/ {print $2}' | sed 's/;//'` lookup ext
   post-down ip rule del lookup ext
   metric 10
sudo ifdown eth1
sudo ifup eth1

Configure Blackberry VPN profile:

Select General IKEv2 VPN server.

IMG_20150628_183531

References:
1. Advanced routing – http://www.rjsystems.nl/en/2100-adv-routing.php
2. Strongswan setup – http://www.math.ucla.edu/~jimc/documents/strongswan-1308.html
3. Strongswan Forwarding and Split Tunnelling – https://wiki.strongswan.org/projects/strongswan/wiki/ForwardingAndSplitTunneling

Posted in Uncategorized | Tagged , , , | Leave a comment

Running Ghost with cPanel on Shared Host via Rails

I have been looking to migrate my WordPress-driven blog over to Ghost but I don’t want to use a VPS since that would require me to configure and maintain every detail. Shared hosting is great in that it is essentially a managed instance – always updated, backed up, and available (depending on the quality of the host, of course). Most importantly, it’s someone else’s problem if there’s downtime.

This post is to document my steps to run Ghost or any other nodejs applications on a cPanel shared web host. While ssh is not required, it makes debugging and testing a little easier. At the time of writing, cPanel does not support nodejs apps natively (https://features.cpanel.net/topic/nodejs-hosting). However, they do support Rails 2 apps, so we can leverage Rails to bootstrap our nodejs app. Note, we cannot just run nodejs applications via command line because port(s) need to be mapped on the account such that yourdomain.com:12345 will be redirected correctly.

For the sake of completeness, I also looked at an alternative method (https://github.com/niutech/node.php) that uses PHP to proxy requests to a node app. That sounds incredibly dirty, though my method is also hacky but arguably less so. Anyway, here it goes.

Outline of steps:
1. Create 2 Rails Applications in cPanel
– one will be used to bootstrap
– second will not be used; we use the assigned port in Ghost or nodejs app
2. Modify Rails app #1 for bootstrapping
3. Install nodejs and npm
4. Upload and configure Ghost
5. Run it and setup redirects as needed
6. …
7. Profit!

1. Create 2 Rails Applications in cPanel
– one will be used to bootstrap
– second will not be used; we use the assigned port in Ghost or nodejs app
ghost_rails1
2. Modify Rails app #1 for bootstrapping

In the bootstrap Rails app ./config/boot.rb, append to the bottom:

1
2
3
4
5
6
7
8
pid = Process.fork
if pid.nil? then
  sleep 5
  exec '(cd ~/node_apps/ghost && npm install)'
  exec '/home/YOURUSERNAME/bin/node /home/YOURUSERNAME/node_apps/ghost/index.js'
else
  Process.detach(pid)
end

note: you may want to immediately exit from the Rails app
note2: screenshot application names maybe different from the commands shown here

3. Install nodejs and npm

1
2
3
4
5
6
7
8
9
10
wget http://nodejs.org/dist/v0.12.3/node-v0.12.3-linux-x64.tar.gz
tar xvf node-v0.12.3-linux-x64.tar.gz
mkdir -p ~/local/bin
cp node-v0.12.3-linux-x64/bin/node ~/local/bin/
rm -rf node-v0.12.3*
echo prefix = ~/local >> ~/.npmrc
wget https://www.npmjs.org/install.sh
chmod o+x install.sh
./install.sh
rm -rf install.sh

note: npm from the node package seems to be statically linked to /usr/local which you won’t have access to, so we install it directly with the prefix option

4. Upload and configure Ghost
– start the bootstrap Rails app from cPanel
ghost_rails4-1

ghost_rails4-2
5. Run it and setup redirects as needed

Posted in Uncategorized | Tagged , , , | Leave a comment