2021-08-17  Deniz Yuret  <dyuret@WS001>

	* TODO:
	- RNNs: ops, layers
	- Transformers: ops, layers
	- Image and text preprocessing layers
	- Huggingface models
	- Checkout pytorch/tf ops and layers for rnns and transformers
	- Get deepmap its own package
	- fileio.jl: needs more work with including bn weights and excluding adam weights.
	- fileio.jl: layers like Linear do not get initialized which makes loading difficult.


2021-03-14  dyuret  <dyuret@login03.kuacc.ku.edu.tr>

	* xnor-lenet:
	- There is a strange state where grad is close to 0 but accuracy is 0.1?

	* training:
	- Not initializing weights is an inconvenience.
	- Still need Train20 and accuracy/nll from Ops20.

2021-02-27  dyuret  <dyuret@login03.kuacc.ku.edu.tr>

	* tf-eval.py: script that evaluates a keras model on imagenet val.
	50K images 4 GPUs 8 mins accuracy=0.7063
	tensorflow/python/keras/applications/mobilenet_v2.py claims 71.8 ???
	https://keras.io/api/applications/ claims 71.3 ???

	Classification Checkpoint|MACs (M)|Parameters (M)|Top 1 Accuracy|Top 5 Accuracy
	--------------------------|------------|---------------|---------|----|---------
	| [mobilenet_v2_1.4_224]  | 582 | 6.06 |          75.0 | 92.5 |
	| [mobilenet_v2_1.3_224]  | 509 | 5.34 |          74.4 | 92.1 |
	| [mobilenet_v2_1.0_224]  | 300 | 3.47 |          71.8 | 91.0 |
	| [mobilenet_v2_1.0_192]  | 221 | 3.47 |          70.7 | 90.1 |
	| [mobilenet_v2_1.0_160]  | 154 | 3.47 |          68.8 | 89.0 |
	| [mobilenet_v2_1.0_128]  | 99  | 3.47 |          65.3 | 86.9 |
	| [mobilenet_v2_1.0_96]   | 56  | 3.47 |          60.3 | 83.2 |
	| [mobilenet_v2_0.75_224] | 209 | 2.61 |          69.8 | 89.6 |
	| [mobilenet_v2_0.75_192] | 153 | 2.61 |          68.7 | 88.9 |
	| [mobilenet_v2_0.75_160] | 107 | 2.61 |          66.4 | 87.3 |
	| [mobilenet_v2_0.75_128] | 69  | 2.61 |          63.2 | 85.3 |
	| [mobilenet_v2_0.75_96]  | 39  | 2.61 |          58.8 | 81.6 |
	| [mobilenet_v2_0.5_224]  | 97  | 1.95 |          65.4 | 86.4 |
	| [mobilenet_v2_0.5_192]  | 71  | 1.95 |          63.9 | 85.4 |
	| [mobilenet_v2_0.5_160]  | 50  | 1.95 |          61.0 | 83.2 |
	| [mobilenet_v2_0.5_128]  | 32  | 1.95 |          57.7 | 80.8 |
	| [mobilenet_v2_0.5_96]   | 18  | 1.95 |          51.2 | 75.8 |
	| [mobilenet_v2_0.35_224] | 59  | 1.66 |          60.3 | 82.9 |
	| [mobilenet_v2_0.35_192] | 43  | 1.66 |          58.2 | 81.2 |
	| [mobilenet_v2_0.35_160] | 30  | 1.66 |          55.7 | 79.1 |
	| [mobilenet_v2_0.35_128] | 20  | 1.66 |          50.8 | 75.0 |
	| [mobilenet_v2_0.35_96]  | 11  | 1.66 |          45.5 | 70.4 |
	|Classification Checkpoint|MACs(M)|Parameters(M)|Top1 Accuracy|Pixel1 CPU(ms)|
	|---|---|---|---|---|
	| mobilenet_v3_large_1.0_224              | 217 | 5.4 |   75.6   |   51.2  |
	| mobilenet_v3_large_0.75_224             | 155 | 4.0 |   73.3   |   39.8  |
	| mobilenet_v3_large_minimalistic_1.0_224 | 209 | 3.9 |   72.3   |   44.1  |
	| mobilenet_v3_small_1.0_224              | 66  | 2.9 |   68.1   |   15.8  |
	| mobilenet_v3_small_0.75_224             | 44  | 2.4 |   65.4   |   12.8  |
	| mobilenet_v3_small_minimalistic_1.0_224 | 65  | 2.0 |   61.9   |   12.2  |

	* pt-eval.py: pytorch evaluation script copied from
	https://github.com/pytorch/examples/tree/master/imagenet/main.py
	mobilenet_v2: Acc@1 71.850 Acc@5 90.334
	Official repo claims 71.878

	MobileNet V2                      71.878          90.286
	MobileNet V3 Large                74.042          91.340
	MobileNet V3 Small                67.668          87.402

2021-02-26  dyuret  <dyuret@login03.kuacc.ku.edu.tr>

	* tf-keras-preprocessing:
	tf.keras.preprocessing.image.img_to_array(img)

	* tf-load-images:
	https://www.tensorflow.org/tutorials/load_data/images

	train_ds = tf.keras.preprocessing.image_dataset_from_directory(
	data_dir,
	validation_split=0.2,
	subset="training",
	seed=123,
	image_size=(img_height, img_width),
	batch_size=batch_size)

	* tf-imagenet-preprocessing:
	https://github.com/tensorflow/models/blob/master/official/vision/image_classification/resnet/imagenet_preprocessing.py

	Training images are sampled using the provided bounding boxes, and subsequently
	cropped to the sampled bounding box. Images are additionally flipped randomly,
	then resized to the target output size (without aspect-ratio preservation).

	Images used during evaluation are resized (with aspect-ratio preservation) and
	centrally cropped.

	All images undergo mean color subtraction.

	Note that these steps are colloquially referred to as "ResNet preprocessing,"
	and they differ from "VGG preprocessing," which does not use bounding boxes
	and instead does an aspect-preserving resize followed by random crop during
	training. (These both differ from "Inception preprocessing," which introduces
	color distortion steps.)

	CODE:
	# For validation, we want to decode, resize, then just crop the middle.
	image = tf.image.decode_jpeg(image_buffer, channels=num_channels)
	image = _aspect_preserving_resize(image, _RESIZE_MIN)
	image = _central_crop(image, output_height, output_width)
	image.set_shape([output_height, output_width, num_channels])
	return _mean_image_subtraction(image, CHANNEL_MEANS, num_channels)

	RESIZE calls:
	return tf.compat.v1.image.resize(
	image, [height, width],
	method=tf.image.ResizeMethod.BILINEAR,
	align_corners=False)


	* resize:
	torchvision.transforms.Resize works for both tensors and images but gives different results.

	ImageTransformations.imresize gives different results.

	https://legacy.imagemagick.org/Usage/resize/ -- ImageMagick has its own resize.

	tf.image.resize: https://www.tensorflow.org/api_docs/python/tf/image/resize
	tf.keras.applications.mobilenet_v2.preprocess_input => imagenet_utils.preprocess_input(data_format=None, mode='tf'): scale pixels to [-1:1]
	tf.image.resize_with_crop_or_pad(i, 224, 224) used by the medium article.

2021-02-24  dyuret  <dyuret@login03.kuacc.ku.edu.tr>

	* tensorflow-eval:
	https://medium.com/analytics-vidhya/how-to-train-a-neural-network-classifier-on-imagenet-using-tensorflow-2-ede0ea3a35ff
	https://stackoverflow.com/questions/37340129/tensorflow-training-on-my-own-image
	https://towardsdatascience.com/hands-on-tensorflow-tutorial-train-resnet-50-from-scratch-using-the-imagenet-dataset-850aa31a39c0
	official resnet script:
	  https://github.com/tensorflow/models/blob/master/official/vision/image_classification/resnet/imagenet_preprocessing.py

	* torchvision-eval:
	official eval script:
	  https://github.com/pytorch/examples/tree/master/imagenet
	Normalization for all torchvision models:
	transforms.Compose([
            transforms.Resize(256),
            transforms.CenterCrop(224),
            transforms.ToTensor(),
            transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
        ])),
	eval command:
	  python main.py --pretrained -e -a mobilenet_v2 /datasets/ImageNet/ILSVRC/Data/CLS-LOC
	  mobilenet_v2: Acc@1 71.850 Acc@5 90.334 (keras version claims Acc@1=71.8 Acc@5=91.0)
	  resnet18: Acc@1 69.644 Acc@5 88.982

2021-02-13  Deniz Yuret  <dyuret@WS001>

	* tf.ZeroPadding2D:

	function zeropad1(x)            # 16ms
	    w,h,c,n = size(x)
	    f = oftype(x, reshape([0 0 0 1], 2, 2, 1, 1))
	    x = reshape(x, (w, h, 1, :))
	    x = conv(f, x; padding = 1)
	    x = reshape(x, (size(x,1), size(x,2), c, n))
	end

	function zeropad2(x)            # 5ms
	    w,h,c,n = size(x)
	    x = cat(x, fill!(similar(x, (1,h,c,n)), 0); dims=1)
	    x = cat(x, fill!(similar(x, (w+1,1,c,n)), 0); dims=2)
	end

	_f = nothing

	function zeropad3(x)            # 1ms
	    global _f
	    w,h,c,n = size(x)
	    if typeof(_f) != typeof(x) || size(_f) != (2,2,1,c)
	        _f = oftype(x, zeros(Float32, 2, 2, 1, c))
	        _f[2,2,1,:] .= 1
	    end
	    x = conv(_f, x; padding = 1, groups = c)
	end

	_x = nothing

	function zeropad4(x) # 3.4ms
	    global _x
	    w,h,c,n = size(x)
	    if typeof(_x) != typeof(x) || size(_x) != (w+1,h+1,c,n)
	        _x = fill!(similar(x, (w+1,h+1,c,n)), 0)
	    end
	    _x[1:w,1:h,:,:] .= x
	    _x
	end

	conv(w, zeropad3(x); groups=128, stride=2) # min=0.2 med=1.4 mean=6.4ms
	conv(w, x; groups=128, stride=2, padding=2)[2:end,2:end,:,:] # min=0.02 med=2.1 mean=1.9ms

2021-02-12  Deniz Yuret  <dyuret@WS001>

	* tf.keras.applications.mobilenet.preprocess_input:
	Takes a (1,224,224,3) image with pixel values in 0:255 Float32.
	Returns a (1,224,224,3) image with pixel values in -1:1 Float32.
	Calls tf.keras.applications.imagenet_utils.preprocess_input(mode='tf')
	tf: will scale pixels between -1 and 1, sample-wise. (means per-image)

2021-02-10  Deniz Yuret  <dyuret@WS001>

	* keras:
	# channels_last (default) corresponds to inputs with shape (batch_size, height, width, channels) while
	# channels_first corresponds to inputs with shape (batch_size, channels, height, width)
	knet default is (w,h,c,n), reverse of channels_first, permutedims(3,2,4,1) of channels last

2021-02-05  Deniz Yuret  <dyuret@WS001>

	* conv: both tf and pytorch seems to perform cross-correlation by default:
	https://www.tensorflow.org/api_docs/python/tf/nn/convolution
	https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html

	* shape: tensorflow vs pytorch weights seem transposed:
	>>> p50 = torchvision.models.resnet50(pretrained=True)
	>>> r50=tf.keras.applications.ResNet50(include_top=True, weights='imagenet', input_tensor=None, input_shape=None, pooling=None, classes=1000)
	>>> pw = p50.conv1.weight.detach().numpy()
	>>> tw = t50.weights[0]
	>>> tw.shape
	(7, 7, 3, 64)
	>>> pw.shape
	(64, 3, 7, 7)

	* images: Julia.Images vs python.PIL.Image
        CUDNN_TENSOR_NCHW,        # 0, /* row major (wStride = 1, hStride = w) */
        CUDNN_TENSOR_NHWC,        # 1, /* feature maps interleaved ( cStride = 1 )*/
	cudnn:nchw => julia:whcn
	cudnn:nhwc => julia:cwhn
	i1=python-image, j1=julia-image
	size(j1) => h,w
	channelview(j1) => c,h,w
	reinterpretc(N0f8,j1) => c,h,w
	np.asarray(i1).shape => h,w,c (python array)
	np.array(i1) => h,w,c Array
	transforms.ToTensor()(i1).numpy() => c,h,w (same as channelview)
